Moonshot AI - Kimi K3 tested

Episode Timeline

1. 0:01

   Introduction to Kimi K3 by Moonshot AI and its pricing and benchmark setup

   • Kimi K3 is a frontier-level open-weight model from Moonshot AI that rivals OpenAI and Anthropic offerings, though most users will access it via API rather than running it locally.
   • API pricing is 30 cents per million tokens for cached input, $3 per million for uncached input, and $15 per million for output, making it significantly cheaper than Claude.
   • The video is sponsored by Moonshot AI, but the host emphasizes the model will be tested with the exact same benchmark suite as always, with no special treatment.
   • The test suite includes performance metrics, a memory test, reasoning benchmarks, OpenAI HumanEval with 164 Python challenges, a Kanban front-end test, sand physics, dungeon crawler, and MCP challenges with Blender and Godot.
2. 2:44

   Performance metrics: prefill and decode speeds on the Kimi API

   • Prefill speed starts around 160 uncached and 240 cached tokens per second and reaches nearly 10,000 cached and 11,800 non-cached tokens per second at large prefill, which is unusual since uncached normally lags behind.
   • The host suspects the benchmark tool is not handling the quantization information correctly, so the cached and uncached numbers may actually reflect two similar runs rather than a true distinction.
   • Decode speed hovers around 43 tokens per second, which is impressive for a 3-trillion-parameter model with 1 million context.
3. 5:21

   Memory test at 256K context and reasoning benchmark results

   • The memory test inserts data at 0, 25, 50, 75, or 100% context depth and asks the model to retrieve it three times per depth, totaling 15 runs, and the model found the data without issue on every run.
   • The host could not run the memory test at the full 1 million context because the API kept timing out, so 256K was used to stay comparable with other models.
   • On the reasoning benchmark the model scored 42 out of 48 for an 88% pass rate, but surprisingly failed twice on medium difficulty while only failing once on hard and three times on expert.
   • OpenAI HumanEval results showed an 86% pass rate overall, a 94% pass rate on answered questions, and a 91% answer rate, indicating the model does not overthink within its 8,000-token limit.

Key Concepts

• Kimi K3— The open-weight frontier model from Moonshot AI that is the central subject of the entire review.
• Moonshot AI— The Chinese company behind Kimi K3 that also sponsored the video and provided API credits.
• open-weight model— The key differentiator that lets users run Kimi K3 on their own hardware, unlike closed models from OpenAI and Anthropic.

Notable Quotes

• So remember this is a 3 trillion parameter model with 1 million context. So the speed is pretty great considering.

  🤯

  — It is genuinely surprising that a model of this enormous scale can maintain 43 tokens per second, revealing how far inference optimization has come.

  Jump to 3:06

• So it's got a 42 out of 48, and that gives us an 88% pass rate. So, a very respectable pass rate overall. But yeah, you can see we're not getting 12 out of 12 on medium. It's fallen down on two reasoning on medium, but then only one on hard and only three on expert.

  💡

  — It overturns the assumption that models fail on harder questions first, showing Kimi K3 paradoxically struggles more on medium difficulty than hard or expert.

  Jump to 4:49

Actionable Takeaways