본문 바로가기
How-to · 2026.10.06

Attach scores and Topics labels to your traces, and you can pull unknown failures out of accumulated logs and feed them back into evaluation datasets and PRs

Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust — AI Engineer · Read the original ↗

This page is a translation of the Korean original. Read the Korean original →

What you get

  • Labels and a topic map surface failure types your existing scores missed
  • Add failure cases to a dataset and see right away whether the next change regressed
  • A coding agent carries the work through from log investigation to code fixes, eval runs and PR writing

When to use it

When a production agent produces too many traces for anyone to review by hand, and you need to find failures your existing scores miss (failed order lookups, broken escalation calls and the like). And when you want to feed those failures back into evaluation datasets and code changes to improve without regressions.

How to do it

  1. Add tracing to a support agent built on the OpenAI Agents SDK and set a baseline with offline evals.
  2. Publish a custom scorer to Braintrust, and set up an online automation, with scope and sampling, that scores incoming traces.
  3. Create a custom facet aimed at failures the default issue classification misses, configure conversation grouping and sampling, then import sample traces.
  4. Use the built-in Loop assistant to query trace data with SQL, examine representative failure cases, and feed those examples back into the evaluation dataset.
  5. Automate log investigation, code and score changes, regression example additions and eval runs with a coding agent equipped with the Braintrust CLI and an improvement skill, and bundle the results into a PR for review.

Try it now

Start by attaching one custom score and one custom facet to a slice of your production agent's logs, and check whether clusters of failures that the default classification hides actually appear. If they do, add representative cases to the evaluation dataset and compare evals before and after a change to verify that the signal leads to an improvement.

Background — what the original covers

This is a record of the Braintrust workshop at the AI Engineer conference. Working from a set of sample traces from a support agent, the presenter argues that observability tools should not stop at showing data but should lead into improvement work, and demonstrates a single flow from tracing to scorer publishing, online scoring, Topics clustering and coding-agent automation. Audience questions turn to deployment options, sampling rates, project scope and access permissions for traces.

#Evals

← All practitioner tips