Treat Your References Like Function Arguments: A Developer's Mental Model for Video-to-Video AI

Most of us met generative video through text-to-video: you write a paragraph, cross your fingers, and get back something that is almost what you meant. If you come from software, that workflow feels oddly unstructured. There's no input contract, no deterministic part, and every retry is a full re-run.

Video-to-video models flip this around, and once I started thinking about them like a function call instead of a wish, my results got noticeably better. Here's the mental model.

Think of a reference-driven video model as something like this:

generate ( source: Video, // motion, timing, camera refs: Image [], // who / what appears prompt?: string, // the glue: which ref goes where mode: " motion " | " swap ", resolution: " 480p " | " 720p " | " 1080p " ): Video Enter fullscreen mode Exit fullscreen mode The key insight: the source video is not a suggestion, it is the bulk of your specification. It already encodes choreography, pacing, framing and camera movement, which is exactly the stuff that is painful to describe in words. Your prompt shrinks from a screenplay to a few lines of wiring.

Motion transfer keeps only the movement and camera of the source and rebuilds everything else from your images. Contract: "same motion, new world."

Object swap keeps the original footage and replaces only the element you point at: a bottle, a jacket, a car, a person. Contract: "same world, one diff."

If you think in diffs, object swap is the git commit that touches one file. Motion transfer is a rewrite that keeps the interface.

Some tools number your uploaded references and let you cite them in the prompt, e.g. @Image1, @Image2, plus @Video for the source. This is basically positional arguments, and the same habits that make function calls readable apply:

@Image1 dances like the person in @Video, wearing the jacket from @Image2 Enter fullscreen mode Exit fullscreen mode replace the bottle with @Image3 Enter fullscreen mode Exit fullscreen mode A few rules that held up in my testing:

Garbage in, garbage out applies more strongly here than with text prompts, because the source video carries so much of the spec.

Generation is slow and costs money, so treat it like an expensive build:

{ "source": "runway-walk-01.mp4", "refs": [ "face_front.jpg", "coat_flat.jpg" ], "prompt": "@Image1 walks like the person in @Video, wearing @Image2", "mode": "motion", "res": "480p", "notes": "identity stable, coat collar drifts at 3s; try second coat angle" } Enter fullscreen mode Exit fullscreen mode Trying it out If you want to poke at this without wiring up any infrastructure, I've been experimenting with Genjutsu, a browser-based video-to-video generator that exposes both modes and the numbered-reference prompting described above, and shows the credit cost before each run. Whatever tool you use, the mental model transfers.

One non-technical note: only use faces, products and footage you have the rights to. "It's just a test" stops being true the moment a clip gets shared.

Stop treating the prompt as the whole program. In video-to-video, the source clip is the spec, the references are the arguments, and the prompt is the wiring. Write it that way and your retries go down.

What patterns have you found for making generative video less of a slot machine? Drop them in the comments.