vLLM says open‑source TPU kernel runs Kimi K3 at 709 tokens a second
The project put that against 450 tokens a second on Nvidia's GB200, and said all 92 of the model's expert layers run in one kernel.
The vLLM project said on Wednesday that a TPU megakernel open-sourced by the inferact team runs Moonshot's Kimi K3 at 709 tokens a second, against 450 on Nvidia's GB200. It credited the inferact team with the work and said a writeup and a repository link sat further down its thread.
SemiAnalysis, which benchmarks inference hardware, reported the same result and put it at 56% better performance than the GB200 NVL72 rack. The firm said the figure showed that Google's work to make its TPU software usable outside Google was, in its words, full steam ahead. Neither account has been reproduced elsewhere.
The technique is a single kernel rather than a faster chip. All 92 of Kimi K3's mixture-of-experts layers run in one Pallas kernel, vLLM said, with weight prefetching that reaches across layer boundaries, so transfers for one layer overlap with computation in the previous one.
Neither post gave the batch size, the chip count or the context length behind the numbers, and a tokens-a-second figure depends on all three. Kimi K3's weights are public, which means anyone with the hardware could in principle check the comparison. Nobody outside the two accounts has yet published one.