Qwen3.8-27B & How to Serve it Fast

Cronologia dell'episodio

1. 0:00

   Qwen releases the 27B model after the massive 2.4T-parameter Qwen 3.8 Max, and its benchmarks surpass Meta's Muse Glimmer and the older Qwen 3.6 27B.

   • Qwen 3.8 Max has 2.4 trillion parameters, making local deployment impractical for most users, so the 27B variant released on Friday is the model everyone was actually waiting for.
   • Qwen's own benchmarks show the 3.8 27B substantially outperforming Meta's Muse Glimmer and the Qwen 3.6 27B across nearly every category, including vision and computer-use tasks where it even beats Opus 4.6 Max.
   • Artificial Analysis scored the model 52 on its intelligence index, just behind GLM 5.2 at 53 and DeepSeek V4 Pro, and far ahead of any other open model runnable without serious hardware investment.
   • On the Agentic Index the model beats GLM 5.2 and some GPT 5.6 variants, which is remarkable for something that can run locally at decent token speeds.
2. 3:28

   The HuggingFace ecosystem already offers many versions of Qwen 3.8 27B, from full BF16 down to 4-bit quants, fine-tunes, abliterated variants, and MLX builds for Mac.

   • Qwen publishes both a BF16 full-resolution version and an FP8 version, while Unsloth has already released 4-bit NVFP4 quantizations that not all GPUs can actually run.
   • Multiple fine-tunes and abliterated (uncensored) versions exist, with the Black Frost AI team among the first to release one, and the MLX community has shipped BF16, 8-bit, and 4-bit builds for Mac users without AMD or Nvidia GPUs.
   • Testing was done on a Dell T2 Pro Max with an RTX Pro 6000 GPU providing 96GB of VRAM, enough to load even the full 16-bit BF16 model without difficulty.
3. 7:40

   Reasoning level dramatically changes token consumption and output quality, with X-high thinking burning tens of thousands of tokens for marginal gains.

   • In the HTML website test, no-thinking mode produced a solid site with zero reasoning tokens, low thinking used about 512 tokens, and medium thinking unexpectedly used fewer tokens than low on this particular task.
   • X-high thinking consumed 17,500 tokens and never finished within the 32k output limit, with runs reaching as high as 22,000 thinking tokens, meaning users need a very large context window and patience.
   • In the pelican SVG test, X-high used 11,000 thinking tokens, medium dropped to under 1,000 with a still-good result, and no-thinking produced a noticeably ugly pelican, making medium the sweet spot.
   • A red dragon variant confirmed the pattern: no-thinking got the bicycle right but the dragon looked childish, medium regressed slightly on the dragon, and X-high used 35,000 total tokens (21,000 for thinking) for a result that was better but still not great.

Concetti chiave

• Qwen 3.8 27B— The newly released 27-billion-parameter open-weight model that is the main subject of the episode.
• local AI inference— The core theme of running large models on your own hardware rather than via cloud APIs.
• reasoning token budget— How many thinking tokens the model uses at different reasoning levels, which dramatically changes speed and quality.

Citazioni rilevanti

• while I've limited it to 32k max tokens out, I've actually run out because I've spent 17,500 tokens on thinking.

  🤯

  — Reveals that the highest reasoning level can consume more than half the output budget on thinking alone, making it impractical for many tasks.

  Vai a 8:45

• I've had the thinking tokens be as high as 22,000 thinking tokens.

  🤯

  — Shows the extreme token cost of X high reasoning, which most users would not expect from a 27B model.

  Vai a 8:45

Conclusioni applicabili