Magnitude: A Self-Compiling Inference Engine That Tunes Kernels to Your Hardware

Y Combinator’s Summer 2025 batch produced Magnitude, an open-source inference engine that takes a different approach to local model performance. Instead of shipping precompiled kernels for broad hardware categories, Magnitude compiles and tunes its kernels on your actual device before a model runs. The project reports up to 2x throughput over llama.cpp — 92% faster decode on Apple Metal and 19% faster on NVIDIA CUDA in its benchmarks — while using 27% less memory per agent session [1][2].

The engine ships as a desktop application for macOS, Linux, and Windows with a bundled CLI. It connects to popular coding agents including Pi, OpenCode, Hermes, Codex, and Claude Code with one click, and exposes an OpenAI-compatible API for everything else [2][6].

How on-device kernel compilation changes the performance equation

Most local inference engines — llama.cpp, Ollama, LM Studio — distribute prebuilt kernels optimized for wide hardware families (for example, “Apple Silicon” or “NVIDIA Ampere”). Magnitude instead compiles kernels on the target machine at first run, then caches the tuned binaries for subsequent launches [1][2]. This means the kernel instructions match your exact chip revision, memory subsystem, and driver version rather than a lowest-common-denominator target.

The project publishes benchmarks for Qwen 3.6 35B A3B at 4-bit quantization with 64k context: on an M4 Pro 48 GB, decode rises from 30 to 57 tokens per second (92% faster); on a DGX Spark, decode moves from 49 to 58 tok/s (19% faster) [6]. Prefill also improves — 9% on Metal, 23% on CUDA. These numbers come from the project’s own test harness; independent reproduction is still early.

What this means for memory and concurrency

Magnitude reports 27% less memory per agent and frees that memory when an agent stops [2][6]. Concurrent sessions share prefix caches, which the project says prevents the slowdown that typically appears when multiple agents hold large KV caches simultaneously [2]. For developers running several coding agents side by side — a common pattern when one agent writes tests while another refactors — this shared-cache design could reduce VRAM pressure on 24–48 GB machines.

Hardware and model support in practice

The engine runs on any Apple Silicon Mac, NVIDIA GPU, AMD GPU, or CPU-only system across macOS, Linux, and Windows [2][6]. There is no fixed minimum specification; smaller machines run smaller models, and more memory enables larger ones [6]. Optimized kernels exist for popular open-weight families (Qwen, Llama, Mistral, and others listed at magnitude.dev/models) [6]. The project writes hand-optimized kernels for these families rather than relying solely on auto-tuning, which is how it claims to beat generalist engines [6].

Integrating with your current agent workflow

Adoption is designed to be low-friction. The desktop app includes a “Connections” pane that detects installed agents (Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, Cline) and links them with one click [2][6]. Anything else works through the OpenAI-compatible API endpoint the app exposes locally [6]. No separate Ollama server or inference runtime is required — Magnitude downloads, configures, loads, and runs models inside its own process [8].

Tradeoffs and open questions

Hacker News discussion surfaced several points worth weighing [7]. First, the project’s speed claims are measured on single-stream decode; real agent workloads involve bursty tool-use loops that change token distribution and may shift bottlenecks to KV cache management or speculative decoding quality. Second, llama.cpp’s Metal kernels already saturate memory bandwidth on many Macs, so a 92% decode improvement on Metal warrants independent verification across model sizes and quantization levels. Third, the UI’s estimated speed numbers have been questioned for accuracy at lower context lengths. Finally, the engine is new (launched September 30, 2026) — ecosystem maturity, model coverage, and long-term maintenance are unproven compared to llama.cpp’s years of community hardening.

Should you try it today?

If you develop locally with coding agents on Apple Silicon or NVIDIA hardware and hit memory or latency ceilings, Magnitude is worth a weekend evaluation. The Apache 2.0 license, zero token costs, and fully private execution (nothing leaves your machine after model download) lower the risk of testing [2][6]. Download the desktop app, connect your existing agent, and run your actual workload — synthetic benchmarks rarely reflect agentic token patterns. Watch the VRAM usage when you run two or three concurrent sessions; that shared prefix cache is the feature most likely to change daily workflows.

Magnitude’s core bet — that on-device kernel compilation beats prebuilt generality — is sound in principle. Whether it delivers consistent wins across the long tail of models, quantization schemes, and agent tool-use patterns is what the next six months of community testing will decide.

Sources

1. Magnitude (YC S25) Launches Self-Compiling Inference Engine, Claims 92% …
2. GitHub - magnitudedev/magnitude: Open source inference engine for …
3. Run the best open models for your machine | Magnitude
4. Local Inference - Magnitude
5. Magnitude’s self-optimizing engine runs · Hacker News | Zeli
6. Magnitude (YC S25) Open-Sources Inference Engine… · AGI Hunt
7. Launch HN: Magnitude (YC S25) - Self-optimizing inference engine for a
8. Magnitude: Open source inference server for local models | Y Combinator