Model to fleet calculator with TibiOS

All prices in US dollars (USD). Estimates for inference only, roughly ±10%. KV cache uses the published attention layout of each large model (MLA latents, linear-attention blocks, sliding windows) and a size-class rule for the rest. Tok/s per user: bytes read per token over memory bandwidth, plus a fixed cost per transformer block (0.1 ms, fitted to public llama.cpp and MLX runs on RTX 5090 / PRO 6000 / 4090, DGX Spark and M3 Ultra: 28 of 35 published figures land within ±30%, the rest within a documented 2.5x), plus two all-reduces per block on multi-GPU boxes (NVLink 0.05 ms, the middle of published TP8 runs on 8x H200 that span 0.13 to 0.33 ms per block; PCIe 0.1 ms); each crossing between your own machines costs the 7 ms round trip TibiOS measured on its wired LAN, and cloud or HGX crossings the InfiniBand / EFA wire time (0.1 ms) of a pipeline. The public Kimi K3 report on two 8x H200 nodes lands at 16.8 tok/s because it runs tensor parallel across the fabric; the page shows that figure next to the estimate. MoE on Metal sits within 2x of the estimate. The aggregate row batches up to 32 concurrent users, as many as their KV caches fit in the memory left after the weights, and is capped by the GPU compute; real serving lands below it. Cost per token uses on-demand prices (AWS us-east-1 after the June 2025 cuts, RunPod Secure Cloud after the September 2026 repricing) or the purchase of your own machines (Macs at Apple list prices, NVIDIA machines at the cheapest online-store prices found in 2026-10) amortised over 36 months, electricity excluded. The on-demand monthly row assumes 12 hours a day; the savings-plan row bills every hour at the 37% discount AWS publishes for a 1-year EC2 Instance Savings Plan. The API reference is the Fireworks.ai serverless list price for the same model class. Prices approximate, as of 2026-10.

(Measured on the TibiOS fleet (2026-10-02): DiffusionGemma 26B-A4B on two Mac minis (M4 + M2 Pro) over LAN: 12.8 tok/s per user, 19.2 tok/s aggregate with 3 concurrent users, TTFT 0.9–4 s, zero swap. Every other figure on this page is an estimate.)

Sources checked on 2026-10-09:AWS EC2 pricing · RunPod pricing · Fireworks pricing · Apple Mac Studio · Tom's Hardware RAM index · DDR5 RDIMM $/GB · H200 server price · llama.cpp on DGX Spark · KTransformers · Kimi K2.5 on EPYC DDR5 · AMD EPYC 9005