Apple Silicon for LLM Inference: Benchmarks and Real-World Numbers

Sovereign buyers in Oman and the wider GCC keep asking the same question about Apple Silicon. The marketing is loud, the YouTube reviews are louder, and the actual inference numbers are scattered across a dozen GitHub discussions and Medium write-ups. This article collects the real-world data for the chips that matter to a ministry, a regulator, or a sovereign bank evaluating an edge appliance: the M3 Ultra and the M4 Max, running open-weight models under llama.cpp Metal and MLX. The pillar guide that frames procurement is the Mac Studio M3 Ultra sovereign edge AI brief; this piece is the benchmark backbone behind it.

Apple Silicon: Neural Engine, GPU, and unified memory

Three architectural choices make Apple Silicon usable for sovereign LLM serving. None are unique to Apple in isolation. Together they describe a class of platform that did not exist in 2023.

Unified memory. The CPU, GPU, and Neural Engine address the same physical pool of LPDDR5X. There is no host-to-device copy across PCIe, no quantisation needed because the model would not fit otherwise, and no juggling of layers between system RAM and VRAM. A 256GB Mac Studio M3 Ultra holds the entirety of a Q4 70B model, the KV cache, and several concurrent contexts in the same allocation surface.

Memory bandwidth. The M3 Ultra ships with roughly 800 GB/s of bandwidth, the M4 Max around 546 GB/s, and Apple's research write-up on the M5 quotes a further step up. Token generation is bandwidth-bound for almost every dense model below 70B, so this is the number that predicts tokens per second more reliably than any TFLOPS chart.

Neural Engine plus GPU. The 32-core GPU on the M3 Ultra runs the Metal kernels that llama.cpp and MLX dispatch. The Neural Engine matters less for token generation than for prompt processing and small-model paths. Apple's MLX research note on the M5 shows up to a 4x speedup on time-to-first-token when neural accelerators are engaged, which moves the bottleneck back to bandwidth.

llama.cpp Metal benchmarks: Gemma 4, Qwen 3.6, DeepSeek R1

llama.cpp with the Metal backend is the reference inference engine for Apple Silicon. It ships in every Hosn Tower image, runs every GGUF quant, and is the operational baseline for sovereign deployments because the entire stack audits cleanly. The community benchmark thread at ggml-org/llama.cpp discussion 4167 aggregates results across the M-series.

Gemma 4 27B at Q4_K_M, M3 Ultra. Roughly 30 to 42 tokens per second of generation on a 4k context, with prompt processing in the 700 to 900 tokens-per-second range. A typical one-page Arabic policy summary completes in under twelve seconds end-to-end.

Qwen 3.6 32B dense at Q4_K_M, M3 Ultra. 25 to 38 tokens per second. Qwen 3.6 30B-A3B (the MoE variant with 3B active parameters per token) is dramatically faster: above 80 tokens per second on the same hardware, because the bandwidth budget per token is much smaller.

DeepSeek R1 distilled 32B at Q4_K_M, M3 Ultra. 22 to 30 tokens per second. The full DeepSeek R1 671B-A37B fits at Q4 in 256GB and runs at 6 to 9 tokens per second, slow but interactive enough for analytical workloads where reasoning quality matters more than latency.

M4 Max class. A 64GB or 128GB M4 Max comfortably runs Gemma 4 12B at 50 to 65 tokens per second and Llama 3.3 70B at Q4 at 8 to 15 tokens per second, useful for single-analyst workstations rather than departmental serving.

Tokens per second by quant level

Quantisation is the single biggest knob a sovereign operator turns. For a fixed model and chip, dropping from BF16 to Q4_K_M roughly triples generation throughput, because each token now reads a quarter of the bytes from memory.

• BF16 (full precision). Reference quality. Required for fine-tuning runs and gold-standard evaluation. Practical for models under 14B on a 192GB+ Mac Studio.
• Q8_0. Nearly indistinguishable from BF16 on Arabic and English benchmarks. Roughly 1.6x faster than BF16 on the same model.
• Q6_K. The recommended floor for legal, regulator-facing, and bilingual translation work where small instruction-following regressions are unacceptable. Roughly 2.2x BF16 throughput.
• Q5_K_M. A sensible default when memory is tight and quality is still a priority.
• Q4_K_M. The production workhorse. Roughly 3x BF16 throughput, with a small but measurable hit on long-context reasoning. Acceptable for general drafting, summarisation, and chat workloads.
• Q3_K_S and below. Reserve for experimentation. Quality degradation on Arabic morphology becomes visible.

The third-party benchmark tracker llmcheck.net is a useful live reference that tabulates real tokens-per-second by chip, model, and quant.

Power per token versus a discrete GPU

For sovereign branch offices, ministry annexes, and regulator field stations, power and thermal envelope are not abstract concerns. A rack-grade H100 server draws 700 to 1000 watts at the wall under sustained inference, requires dedicated cooling, and rarely fits inside a 12U cabinet that already houses networking and a UPS. A Mac Studio M3 Ultra under the same load sits at 200 to 270 watts and runs silently in an office cabinet.

On tokens-per-watt for the model sizes that fit (8B to 70B dense, MoE up to 120B), Apple Silicon is two to four times more efficient than a comparable discrete-GPU configuration. That ratio reverses above the 70B-dense threshold, where memory bandwidth limits push Apple platforms below useful concurrency rates and the institutional H100, H200 and RTX 6000 comparison applies. The right answer for a national ministry that serves a thousand staff is a rack; the right answer for a regulator with twenty analysts in a satellite office is a Mac Studio appliance, sometimes paired with a small edge GPU appliance for redundancy.

If your team is choosing between an Apple-based edge appliance and a rack build for an Omani sovereign deployment, email [email protected] for a one-hour briefing. We benchmark on the chips you would actually buy, with the models you would actually run.