How to Build Things with Jev & OpenJevs

Linha do tempo do episódio

1. 0:00

   Introduction to building a model router with Jev and the models used locally and in the cloud.

   • Jev is a system-one model from Typesafe AI that does not generate text but returns typed answers with probabilities for choice, score, and null questions.
   • The router runs locally as an endpoint and uses a mini CPM 5 2B model locally, Qwen Image 2.1 for images, and DeepSeek V4.1 Flash via OpenRouter for cloud tasks.
   • Jev charges nothing for output tokens and only 4 cents per million input tokens, with responses typically in the 70 to 500 millisecond range.
   • The three question types map neatly onto routing: choice for categorization, score for difficulty, and null for privacy gating.
2. 3:39

   Demonstration of the router deciding between local and cloud models based on Jev's judgments.

   • A simple greeting was routed to the local mini CPM model with zero difficulty and no private information, decided by Jev in 331 milliseconds.
   • A Python code request was routed to DeepSeek because the choice category returned code with near 100% probability, and difficulty increased above chitchat.
   • The choice question includes classes like chitchat, simple question, rewrite/summarize, code, reasoning/analysis, and image, each with defined criteria.
   • The score question uses levels from trivial to hard, where trivial means a small 2B model handles it easily and hard requires frontier-level reasoning.
3. 8:00

   Privacy gating and the architecture of the router with FastAPI, SQLite, and OpenAI-compatible endpoints.

   • A fake API key triggered the privacy null question, which returned 0.96 probability of private data and forced the request to stay local.
   • The architecture uses a Next.js UI sending to a local FastAPI server that makes one Jev call for decisions, then routes and streams from the winning model.
   • Lanes are preconfigured with an order of preference, and conditional rules include routing to web-enabled models if difficulty exceeds 6 or to local models if the privacy flag is above 0.5.
   • Everything is logged to SQLite, including Jev decisions, messages, and generated images, with health checks and OpenAI-compatible endpoints for all models.

Conceitos-chave

• Jev— The core system-one model from Typesafe AI that powers the entire routing logic without generating text.
• model router— The main project built in the episode, deciding which model handles each request.
• typed questions— The three question types (choice, score, null) that Jev answers with probabilities.

Citações notáveis

• And remember, the cool thing for this is it doesn't charge you anything for output tokens at all, and it only charges you 4 cents per million input tokens.

  🤯

  — Reveals an unexpectedly low pricing model that makes routing decisions nearly free compared to standard LLM calls.

  Ir para 2:35

• A nice way of thinking about this is that you can kind of think of it as an if statement that understands language.

  💡

  — Reframes a sophisticated AI model as a simple programming construct, making the concept instantly graspable.

  Ir para 2:35

Ações a tomar