2026 · in progress
MiniInfer
understanding inference by building the machinery myself.
why
i could use an LLM API without knowing how it works, the same way i can drive without knowing how an engine works. but "the model generates the next token" stopped being a satisfying sentence. i didn't understand inference, so i'm building a smaller version of it.
the problem
inference looks simple from the outside: weights in, text out. inside, every token passes through embedding lookup, attention over everything generated so far, and a final projection into vocabulary-sized logits. the interesting part isn't any single step, it's what generating token number 500 costs that token number 5 didn't.
the first thing that surprised me: the bottleneck isn't arithmetic. it's
architecture
MiniInfer is starting in Python, where iteration is cheap and every tensor can be inspected. the plan is a small, legible runtime: tokenizer, embedding lookup, transformer blocks with a KV cache, logits, and sampling, each stage observable on its own.
current state is early: the runtime is bootstrapped and the first experiments are matrix-level building blocks. the decode loop doesn't exist yet, which is honest for a project whose point is the journey through each stage.
what i learned
so far, mostly how much sits underneath one API call: tokenization, the attention mechanism's memory footprint growing with context, and why serving systems obsess over batching and caching rather than raw model size. the learning notes are becoming writing as they solidify.
what broke
nothing dramatic yet. the project is young enough that the interesting failures, the shape mismatches, the off-by-one in the cache, the sampling bug that produces fluent nonsense, are still ahead. they'll be documented here when they happen.
what's next
a working decode loop with a real KV cache, then sampling strategies, then measuring where time actually goes. lower-level work, CUDA, and possibly a second implementation in a systems language, follow once the Python version has taught me what to look for.
all projects