A million vectors, still instant: the search engine we wrote in Rust
Finding the nearest meaning among a million cached answers, in under a millisecond, without a single external dependency. A look at the pure-Rust HNSW engine underneath Crowkis.
Every cache hit starts with a search problem: given a new question's meaning, find the closest question we've already answered, out of possibly millions. Do it slowly and the cache is pointless; the model call would've been faster. So the search has to be effectively instant, and it has to stay instant as the cache fills up.
A navigable small-world graph: a few hops land on the nearest neighbour, even at a million points.
We wrote ours from scratch, in Rust, with no C libraries bolted on. That sounds masochistic until you see the payoff: the search understands that a cache entry isn't just a vector, it carries a trust score, a freshness stamp, an intent template. Owning the engine means the reuse checks happen right next to the data, not across five foreign API calls per lookup.
In our benchmarks, searches stayed sub-millisecond while recall held near-perfect into the millions.
The result: at a million cached meanings, a search still returns in well under a millisecond and almost never misses the true nearest neighbour. Because the whole engine is pure Rust with the model bundled in, the entire thing ships as one small container, nothing to download at runtime, nothing external to trust.
The bottom line
Fast, accurate, self-contained, pick three. The cache in the hot path of every LLM call can't afford to be slow, and it can't afford to hallucinate a neighbour. Owning the search engine is how you get both, and it's why Crowkis fits in one file.