Plug your own scenarios straight into several models and run them — you see the real differences in your app that benchmarks cannot tell you
The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft — AI Engineer · Read the original ↗
This page is a translation of the Korean original. Read the Korean original →
What you get
- See for yourself where a model breaks down: character counting, grounding, images, or tool calls
- Distinguish gaps a prompt can close from gaps it cannot (image understanding)
- Bootstrap a test set with ASSERT and use DSPy to get prompts that even cheaper models pass
When to use it
Use it when you need to evaluate whether a model in a RAG application makes up answers from its own weights instead of grounding them in sources, and when, since model access can be cut off, you need to make sure the application still works when pointed at a different model.
How to do it
- Clone the repository and sync packages with UV, or if you have no Python, create a GitHub Codespace, add your .env, and start.
- Change the model variable comment at the top of each example file and run the same prompt on four models to see the differences with your own eyes.
- Close the gaps on failing tasks with prompt engineering that splits steps and makes the model reason, but assume that image understanding cannot be closed with prompts.
- Where arguments are clear, as in tool calls, run large numbers of test cases with string-comparison evaluation; where output is ambiguous, move up to LLM-as-a-judge, which gives pass/fail together with a reason.
- If building evaluation from scratch is hard, use ASSERT to generate behavior categories and a test set from a spec YAML, and use DSPy to declare constraints and optimize the prompt.
Try it now
Put a model variable at the top of the prompt file you use now, then change only the comment and run the same input on two or three models. You will see right away whether the point where answers diverge can be closed with a prompt or is a limit of the model itself. If it is an agent, it is better to log tool call counts and whether calls ran in parallel, and compare them.
Links
Background — what the original covers
This is a hands-on workshop that Microsoft's Pamela Fox gave at AI Engineer World's Fair 2026. Running the same examples in turn on four models deployed on Foundry, it raises the difficulty from a simple single call to RAG, image input, tool calls, loops, and a LangChain v1 agent. The second half moves on to string checks and an LLM judge, ASSERT, recently released by Microsoft Research, and DSPy prompt optimization.