본문 바로가기
How-to · 2026.10.19

Draw the process as a DAG and turn each node's check into an evaluation — you can see which steps to hand to an agent and which steps humans still hold

How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase — AI Engineer · Read the original ↗

This page is a translation of the Korean original. Read the Korean original →

What you get

  • You get a criterion for which steps can be automated without violating policy
  • You can explain quality with statistics and traces instead of a feel for the model
  • Human involvement is reduced step by step: oversight → sampling → exception handling

When to use it

Apply it to policy-based processes such as commercial lending (KYC, financial review, credit memo, risk review, documentation, decision), token listing workflows on a regional exchange, and work with approval and verification gates such as regulated software deployment flows.

How to do it

  1. Write the policy first in must/may/should language, and separate the "build state" that defines and changes policy from the "run state" that keeps running.
  2. Draw the workflow as a DAG to expose steps, dependencies, and mandatory gates, and find the reviews that can run in parallel.
  3. Design the inside of each node as a loop of work → check → feedback (DCG), and secure operational visibility across the whole process so you can see exception flows and bottlenecks.
  4. Build a representative dataset to establish the definition of "good," and use repeated evaluation runs and runtime traces as the trust mechanism.
  5. For each node, ask whether policy comparison, loop boundaries, tracing, and human fallback are possible to choose the steps the agent takes on, and raise autonomy in the order of full oversight → sampling → exception handling.

Try it now

Pick one workflow you are responsible for now, draw its steps and mandatory gates as a graph, and put four questions to each node: can policy comparison, loop boundaries, tracing, and human fallback be done? The answers show which steps suit an agent and which must remain with human judgment. It is also worth checking the point that a decision node is itself a check against policy and so must be held by a person.

Background — what the original covers

This is a talk by Evan Kormos of Coinbase on how to bring agents into work in heavily regulated environments such as finance. He treats an organization's policy as the original system prompt, and explains work as a forward-only graph with loops that repeat inside it. Using code review and commercial lending as examples, he points out where to put agents and what to use to confirm trust.

#Agents#Safety

← All practitioner tips