Treat video generation as next-frame prediction and judge it on physics and geometry, and you measure how well a model understands the world, not whether it suits someone's creative taste
You Can't Tell What's Real Anymore. Here's What Comes Next - Cristóbal Valenzuela, Runway — AI Engineer · Read the original ↗
This page is a translation of the Korean original. Read the Korean original →
What you get
- Gain an evaluation axis that scores model quality on physics, geometry and object behaviour instead of 'do I like it'
- Quantify model progress by using whether people can tell real footage from generated footage as the metric
- Preview a new kind of interface: interactive screens that respond to clicks and resizing without a 3D engine
When to use it
The talk cites two use cases: world models serving as the 'brain' of robots and embedded systems, and coding agents that reason about problems visually instead of in text (for example, working through a request to build a 3D model of a bicycle visually).
How to do it
- Frame video as a next-frame prediction problem and scale it up with a large enough network and enough compute.
- Build evaluation benchmarks on physics, geometry and object behaviour, not on 'do I like it' creative preference.
- Measure whether people can tell real from generated video to quantify the model's level.
- Let users intervene at any point during prediction and inject actions, turning the model into a real-time interactive simulator.
- Train it on clicks and manipulations of an interface so on-screen content opens as a real-time video portal, and connect it to a coding agent so the agent can think visually.
Where I agree or hold back
The idea of building evaluation benchmarks on verifiable axes such as physics, geometry and object behaviour, rather than preference votes, transfers directly to domains beyond video. The forecasts that 'everything will be simulated' or that operating-system screens will be streamed from a video model are another matter: the speaker himself called them early research, so hold judgement until they are released.
Links
Background — what the original covers
At AI Engineer, Runway co-founder Cristóbal Valenzuela traces the path from 128x128-pixel clips in 2019 to today's video models. He argues that if you scale the principle cinema discovered some 120 years ago, that stringing frames together creates motion, into a large prediction problem, the model learns how the world works. He also presents Continuum, a research prototype that pushes this into real-time interaction, along with Runway's robotics lab.