본문 바로가기
Case · 2026.10.12

Define agents as YAML workflows, not free-running loops — per-step permissions show up in code review and runs become predictable

Agents in Orbit: Workflows, Not Autopilot — Eduardo Simon & Javier Casal, Flywire — AI Engineer · Read the original ↗

This page is a translation of the Korean original. Read the Korean original →

What you get

  • You can narrow network, tool, skill and MCP permissions step by step, for example letting the model read code but not change it
  • Locking the predictable parts into code makes runs more stable and cheaper than open sessions
  • Agents can run on in-house deployment infrastructure without sending sensitive payment data outside

When to use it

Use it for repetitive engineering work such as automated merge request code review, translation syncing, fixing vulnerable dependencies, investigating production errors in Sentry and ticket triage. Also apply it to large changes that span many repositories, such as migrations, library updates and deprecations.

How to do it

  1. First write a repeated, well-defined task as a skill, try it for a day, and check whether it is worth automating.
  2. Once its value is confirmed, move the task into a workflow: keep the predictable parts as deterministic code and call the model only where judgment is needed.
  3. Define the agent in a single YAML file in the repository and put it through the same merge request review as any other code.
  4. For each LLM step, declare in the YAML which network, tools, skills and MCP servers it may access (for example, a code review step can only read and search, not edit or push).
  5. Enforce a strict JSON schema on every LLM response, and collect production run traces into a golden dataset to compare and improve prompts and models.

What to borrow

Rather than trying to evaluate an agent as a whole, isolate a single LLM step and feed it the state from past runs; that approach, along with building run traces into a golden dataset, is worth adopting right away. Take the speakers' admission at face value: new workflows go into production after checking only a handful of real cases, and there is no online evaluation yet. Do not read the reported success rates as a measure of answer quality.

Background — what the original covers

Flywire, which processes international payments, has about 400 engineers running several language stacks under regulations such as PCI and SOC 2. It pulled together AI experiments scattered across the company into an AI platform team and built Orbit, an internal platform for eliminating repetitive work, on top of its existing in-house PaaS. Alongside their results, the presenters say plainly that sandboxing and online evaluation are not yet finished.

#Agents#Safety

← All practitioner tips