본문 바로가기
How-to · 2026.10.20

Don't settle for a decision model's default 0.50 threshold — tune it on your own labelled data and the same model catches failures it used to miss

How To Use Decision Models for Evals — Hamel Husain (evals) · Read the original ↗

This page is a translation of the Korean original. Read the Korean original →

What you get

  • Recover classification performance that the default threshold throws away
  • Set minimum precision and recall first, so unusable models are screened out early
  • Use a calibration check to tell whether scores can be read as confidence

When to use it

Use it when you need to automatically find traces that contain failures (failed bookings, missed handoffs to a human, and so on), such as 100 conversation and action logs from an apartment-leasing agent. For classifiers that must hold down both false positives and misses, such as automated alerting systems, set minimum precision and recall thresholds first to screen models.

How to do it

  1. Refine the judge prompt until you are satisfied with it, and use a separate judge for each failure mode.
  2. Set minimum precision and recall thresholds for your use case, and drop any model that falls short.
  3. Give a coding agent the labels and stored scores, tune the decision threshold with cross-validation, and compare it against the 0.50 default.
  4. If you want to use scores as confidence, check with a calibration chart whether predicted probabilities match actual failure rates, and measure whether Platt scaling helps.
  5. Weigh tuned performance against cost and latency to pick the right model for the workload.

Try it now

If you already have labelled traces and the scores a model produced, you can run the threshold-tuning prompt right away, with no new API calls. Check how much precision and recall change compared with the 0.50 default, and how many splits clear the minimum bar you set. Rather than using F1 as is, first decide which is more costly: false positives or misses.

Background — what the original covers

A wave of decision models — Jev, Perplexity Decider, Liquid d1, Cloudflare Clef Flash, OpenAI Decisions — is drawing attention as a cheap option for automated evaluation. The author ran 100 traces from a leasing agent through each model with the same evaluation criteria, comparing how well each picked out logs containing failures, on both performance and API cost. It is a record of applying classifier fundamentals that practitioners have always used — threshold setting and probability calibration — directly to evaluation work.

#Evals

← All practitioner tips