Shackled to the Machine: A DeepSeek-R1-8B Record on 4× Raspberry Pi 5

A dense 8-billion-parameter reasoning model (DeepSeek-R1-Distill-Llama-8B) sustains 8.32 tok/s decode on four Raspberry Pi 5 boards — CPU-only, bit-exact, the best published throughput for this model on Pi 5 hardware with distributed-llama: 29% above the prior four-node figure (6.43 tok/s) and roughly six times a single board. The more useful finding, however, is where that improvement comes from. On this workload our twelve source-level changes — the same kernel fusions and framework patches that lifted a 30B Mixture-of-Experts model by 15.2% on identical hardware — contribute approximately zero. Dense decode is bound by memory bandwidth, and the layers that move it are the firmware, the operating system and the thread configuration, not the compute kernels. This is a condensed technical write-up of our report, companion to Pushing Four Raspberry Pis to the Memory Wall; the full paper — every table, figure and raw log — is one click away.

Results at a glance

The question this study asks

Our companion report optimised a 30B Mixture-of-Experts model (Qwen3-30B-A3B) on this same four-node Pi 5 cluster and measured a bit-exact +15.2% decode improvement attributable to source-code changes alone. The natural follow-up: do the same optimisations generalise to a dense model? DeepSeek-R1-Distill-Llama-8B is the obvious target — a widely used open reasoning model, commonly associated with running language models on a Raspberry Pi, with published distributed baselines to compare against.

The answer is no — and the reason is the substantive finding. We frame inference performance as a three-layer stack — hardware and firmware, the operating system, the application software — and argue that the productive move is to profile the binding constraint and then tune the layer in which it lives. For the MoE model that layer was the application kernels (3B active parameters per token make decode compute-bound). For this dense 8B model, every decoded token reads all eight billion parameters: decode is bandwidth-bound, the kernels make no measurable difference, and the layers that pay are the firmware and the operating system.

Three hard constraints

The study preserves the constraints of the companion work — which rules out the lossy speedups and makes every surviving number deployable with zero quality risk:

• Bit-exact output — the SHA-256 of the generated token-id sequence matches a fixed reference (seed=42, temperature=0). Every configuration is gated on this; a config that fails the hash is rejected.
• No CPU overclocking — the silicon runs at its nominal 2.4 GHz.
• No model change, no quantisation reduction — DeepSeek-R1-Distill-Llama-8B Q40 is fixed.

System design

Per node: Broadcom BCM2712 (4× Cortex-A76 @ 2.4 GHz, ARMv8.2-A, NEON with dot-product); 16 GB LPDDR4X @ 4267 MT/s (≈17 GB/s theoretical); integrated Gigabit Ethernet (intra-cluster latency 0.226 ms); Debian 13, kernel 6.18.29 aarch64, 16 KB base pages, performance governor. No usable accelerator — the V3D GPU lacks compute shaders for this workload and there is no NPU. One root node plus three workers, tensor-parallel synchronisation per transformer layer.

A point worth stating plainly: the 16 GB RAM is not a factor. The 8 GB and 16 GB Pi 5 share the identical LPDDR4X interface and bandwidth — capacity, not bandwidth, is what differs — and the 8B Q40 model (~1.6 GB per node) fits comfortably in 8 GB. The extra RAM is headroom only and gives no decode advantage over the 8 GB-based prior record.

Model: DeepSeek-R1-Distill-Llama-8B, Q40 weights (4-bit) with Q80 activation and synchronisation buffers; dense — eight billion active parameters per token, against three billion for the MoE companion. The Q40 model is 6.32 GB on disk, and each decoded token reads approximately 1.1 GB of weights per node.

Software: distributed-llama on the pi5-cluster branch, built with the Cortex-A76 flag set (-O3 -flto -ffast-math -funroll-loops -mcpu=cortex-a76 -march=armv8.2-a+fp16+dotprod+rcpc); workers at nthreads=4 (the clean-room optimum), jemalloc preloaded.

Methodology

The decode metric is the CLI Prediction tokens/s reported by dllama inference — the same metric used by the public figures — measured with standard prompts. Decode is steady-state and essentially independent of the prompt, which we verify directly. Benchmarks use n = 20 runs after two discarded warm-ups; firmware, governor and temperature are checked before each measurement (2.4 GHz, no throttling, 53–55 °C). Every configuration is gated bit-exact. The benchmark and verification scripts are committed in the repository (deploy/scripts/).

The headline: a clean-room thread sweep

Following the companion report, we re-measured decode in a clean-room configuration — observability stack disabled — sweeping the per-node thread count. Decode increases monotonically with thread count up to the core count, with non-overlapping 95% confidence intervals:

The fourth thread is the optimum because its compute gain outweighs the network-receive contention it adds on the shared core. We tested receive-flow steering (RPS) directly: 8.360 tok/s with RPS on versus 8.359 with it off — no measurable effect, because this NIC has a single receive queue whose hardware IRQ and RPS target both land on CPU 3.

Disabling the observability stack contributes only about two per cent here, in contrast to roughly five per cent on the compute-bound MoE model — dense decode is largely insensitive to the modest CPU draw of metrics agents, because it is not waiting on CPU.

The honest part: our source changes contribute nothing

To attribute the improvement we rebuilt clean upstream distributed-llama (git clone, default make) on the same four boards and ran a same-session clean-room A/B at the four-thread optimum:

A stronger control removes the source variable entirely: the production binary that served the headline sweep is byte-identical to a clean upstream checkout (git e0c5973, default Makefile, matching SHA-256 on all four nodes). The 8.32 headline is therefore reached by the stock upstream binary itself.

So where does the 29% over the 2024 record come from? It is the cumulative effect of the full system we built — upstream progress between versions, a newer Linux kernel, SDRAM bank-interleave firmware, scheduler and network tuning, and the four-thread configuration — against an older, untuned baseline. Of the components we could isolate on this workload, the thread count and the firmware are the movers; the compute-kernel rewrites and NIC tuning, which drive the compute-bound MoE result, measure ≈0 here. We report this null result prominently because it is the substantive finding of the study.

The firmware lever: the one change that touches bandwidth

The bootloader EEPROM enables SDRAM bank-low interleaving on every node. This is the single change in the stack that affects memory-access throughput, and therefore the only one that can move a bandwidth-bound dense decode. It is bit-exact, persistent, and present in both arms of the attribution — it raises the floor of both builds above the prior four-node figure without being a source-code contribution.

Where the software does pay: prefill

The same optimisations that leave decode unchanged roughly double prefill, because prefill is compute-bound (batched matrix multiplications with high arithmetic intensity):

For long-context and retrieval-augmented workloads, where prefill is a substantial fraction of latency, this matters even though it does not change the decode figure.

The bottleneck, quantified

Decomposing each decoded token into on-node compute-plus-memory (Pred) and network all-reduce (Sync):

With approximately 1.13 GB of weights read per node per token, sustained traffic is ≈10.1 GB/s per node against a vendor ceiling near 17 GB/s — the practical LPDDR4X limit. ARM PMU sampling during steady-state decode gives 0.81 instructions per cycle with the cores far from saturated: memory-stalled execution, not a compute limit. A bandwidth-bound workload cannot be accelerated by the application kernels — only by the hardware and operating-system layers.

Decode is not a constant: the KV-cache law

The 8.32 headline is the short-context rate. distributed-llama emits per-token timing, so a single long generation traces the entire decode-rate curve. Across three very different 5,000-token generations — a mathematical proof, a B-tree implementation, and a logic puzzle — decode falls almost linearly with context position as the KV-cache grows; the three curves overlap within 0.17 tok/s, so the rate is a function of context length, not content:

Per-token wall time is linear in context position with near-perfect agreement: t(L) = 116.73 + 0.01740·L ms/token (R² = 0.9993). The slope is the cost of reading one more token's KV-cache (all 32 layers) each step — an implied ≈3.8 GB/s on the same LPDDR4X bus. The network share stays at 16–18% across the whole curve, so the slowdown is entirely on-node KV traffic: the bandwidth wall deepens with context. The practical consequence: a long reasoning trace, which R1-distilled models routinely produce, runs at the integral of this curve — about 6.2 tok/s averaged over 5,000 tokens, below the short-context headline. We also re-ran the thread sweep while generating 2,500 tokens: four threads is optimal at every context length (a flat ≈13% over three threads), so the headline configuration is robust to generation length.

Operating-system tunings, one parameter at a time

We evaluated the available software and operating-system parameters one at a time, each gated bit-exact against the baseline. None changes dense decode — the expected outcome once the workload is at the bandwidth limit on an already-tuned platform:

What did not work

In keeping with the companion's reporting norms, the rejected candidates and why:

We began from the expectation that the source changes which accelerated MoE decode would also accelerate dense decode. The clean-upstream attribution refuted that expectation, and the bottleneck decomposition explains why. On a bandwidth-bound workload the productive use of engineering effort is prefill, speculative decoding and hardware — not the decode kernels.

Adaptive speculative decoding: the remaining lossless lever

Because decode is bandwidth-bound, the only lossless way to increase it is to emit more than one token per weight read. We implemented prompt-lookup speculative decoding within distributed-llama, entirely in the root inference loop — it reuses the engine's existing batched forward, so the workers require no recompilation. Each step drafts k tokens from the most recent matching n-gram in the generated sequence, verifies them in one batched forward, and accepts the longest prefix equal to the model's own argmax. The output is identical to greedy decode, verified by SHA-256.

A fixed large draft hurts on non-repetitive text (a rejected draft costs roughly three times a single token), so the draft length is adaptive: a counter grows when drafts are accepted, shrinks when they are not, and falls to zero — batch size one, no penalty — when the text stops being predictable, with a periodic probe to detect repetitive passages again.

On DeepSeek-R1 specifically, the reasoning phase dominates the output and consists of novel text with little n-gram overlap, so on a realistic retrieval prompt most of the generation remains neutral. It is an opt-in accelerator, not a uniform speedup — and we report it that way.

Dense versus Mixture-of-Experts: a controlled two-point measurement

Placing this dense 8B model alongside the 30B MoE model from the companion report — same boards, same optimisations — isolates why software helps in one case and not the other:

The MoE model activates three billion parameters per token, so each decoded token reads far fewer weight bytes: decode is compute-bound and kernel work pays. The dense model reads all eight billion: decode is bandwidth-bound and the same work yields approximately zero. Measuring both models on identical hardware isolates this distinction and shows that the highest-leverage layer is workload-dependent.

Optimisation in layers

The results organise into a layered account of where inference performance comes from — a restatement of the roofline view, not a new idea, but one worth making concrete:

The rule that organises every result here: profile the binding constraint, then optimise the layer where it lives. For dense decode that layer is the system — the fourth thread alone moves decode 16%, and the firmware interleave plus a newer kernel raise the floor to 29% over the prior record — while rewriting the application kernels makes no measurable difference. We expect the method, if not the prescription, to carry to other inference software (GPU model serving, detection models); validating that is future work.

Caveats and limitations

• The 29% headline spans software versions and an older, untuned baseline. The prior figure (6.43 tok/s) is a 2024 result on distributed-llama v0.12.2 with 8 GB boards. The 8 GB and 16 GB Pi 5 share the same LPDDR4X bandwidth and the model fits in 8 GB, so the RAM SKU does not affect decode — but the comparison is system-vs-system, not a controlled single-lever measurement. The controlled measurements are the thread sweep and the clean-upstream A/B inside this study.
• The three- to four-thread optimum reversed between an earlier in-house measurement and this one. The change coincided with a kernel upgrade (to 6.18.29) that we could not isolate, as no safe downgrade was available; we attribute it tentatively.
• The headline is platform- and configuration-specific. The 29% is system-level engineering — platform, thread and firmware configuration, and upstream integration — rather than kernel rewriting. Both are our work; only the system layer moves a bandwidth-bound workload.

Conclusion and future work

DeepSeek-R1-Distill-8B sustains 8.32 tok/s short-context decode on four Raspberry Pi 5 boards, bit-exact, the best published figure for this model on this hardware with distributed-llama — and the decode rate falls predictably with context, to about 5.0 tok/s at 5,000 tokens, following a one-parameter KV-cache law. The deeper result is the attribution: on a dense, bandwidth-bound workload, source-code optimisation contributes nothing measurable, and the system layer — firmware, kernel, thread placement — is where the throughput lives. The contrast with the compute-bound MoE companion, where the same source changes yield +15.2%, frames the bandwidth limit as a controlled comparison rather than an anecdote.

Beyond the adaptive speculative decoder — which converts repetitive or context-quoting output into up to 1.96× without any quality loss — dense-decode throughput on this hardware is a function of memory bandwidth, and further gains require hardware changes: faster memory, a faster interconnect, or additional nodes. More broadly, inference performance is a three-layer stack — hardware and firmware, operating system, application software — and the productive optimisation is to profile the binding constraint and tune the layer in which it lives. On this workload, that layer is the system rather than the kernels.

Resources

• Full paper, raw logs & data: Zenodo, DOI 10.5281/zenodo.20411979 — includes the clean-room sweep and attribution logs.
• Companion paper (MoE on the same cluster): Pushing Four Raspberry Pis to the Memory Wall (DOI 10.5281/zenodo.20357376).
• Code: github.com/hellomatik-org/distributed-llama (branch pi5-cluster, benchmark scripts in deploy/scripts/).
• The public reference we compare against: b4rtaz/distributed-llama discussion #162.