What I Learned Building Tiny Desk AI: The Challenges of Five-Person Video Generation

AI video generation has become increasingly capable of turning a single portrait into a realistic animated performance. However, generating a video with five different people, consistent appearances, and precise camera transitions introduces a completely different set of engineering challenges.

I'm currently developing Tiny Desk AI, a web application exploring personalized concert-style videos generated from user-uploaded photos.

One of the most interesting experiments has been creating a five-person music performance in which every performer is assigned a different reference image.

In this article, I'll share some of the technical problems I encountered and the approaches I've been exploring to improve the results.

A user uploads five portrait photos, and each photo is mapped to a specific performer in a music performance.

The five roles include a featured lead singer, three supporting vocalists, and a saxophone player.

The challenge is making sure that each person appears in the correct role throughout the entire video, including close-ups and camera transitions.

For example, this is the intended mapping:

const characterMapping = { leadSinger: " picture1 ", vocal1: " picture2 ", vocal2: " picture3 ", vocal3: " picture4 ", saxPlayer: " picture5 ", }; Enter fullscreen mode Exit fullscreen mode The mapping looks simple in code, but a generative video model may not reliably maintain those assignments throughout the output.

One of the most noticeable issues occurs when multiple characters appear in the same scene.

A model might correctly generate the first three performers while accidentally assigning the wrong identity to the saxophone player.

Another common issue is identity drift across camera cuts. A performer can initially appear with the correct face and hairstyle, then gradually develop different features in a later shot.

The prompts I tested explicitly required fixed character assignments and prohibited identity duplication.

However, stronger instructions alone did not guarantee consistent results.

This led me to investigate more structured character references and shot-specific generation workflows.

For one experiment, I used a reference performance lasting approximately 13.59 seconds.

The video contained three principal segments:

The initial generation configuration requested a 15-second result at 720p resolution.

{ "model": "seedance-2.0-mini", "duration": 15, "ratio": "9:16", "resolution": "720p", "generate_audio": true } Enter fullscreen mode Exit fullscreen mode One generated result introduced an additional camera transition during the lead singer's segment and extended the final saxophone performance beyond the original reference.

The difference between the reference duration and requested output duration was one possible contributing factor.

I learned that preserving temporal structure requires more than describing the intended performance. It is also necessary to inspect whether the generation workflow can respect exact shot boundaries.

An approach I'm exploring is representing the reference video as a sequence of predefined shots.

const referenceShots = [ { start: 0, end: 5.292, performers: [ " vocal1 ", " vocal2 ", " vocal3 " ], }, { start: 5.292, end: 11.583, performers: [ " leadSinger " ], }, { start: 11.583, end: 13.593, performers: [ " saxPlayer ", " vocal1 " ], }, ]; Enter fullscreen mode Exit fullscreen mode This representation provides a useful source of truth for validation.

It helps answer questions such as which identities should appear in a particular shot, when the camera should switch, and whether a recurring performer is represented consistently.

For example, the keyboard performer appears in the opening segment and becomes partially visible again in the final saxophone shot.

That performer must retain the same assigned identity in both appearances.

The metadata does not force a video model to follow these constraints automatically. It provides a foundation for evaluating results and potentially splitting generation into smaller, more controllable tasks.

Another lesson involved the difference between facial identity and complete character appearance.

In some generated results, the replacement person had recognizable facial features but retained the original performer's clothing or hairstyle.

For a user expecting their uploaded character to appear naturally in the concert, this can create an inconsistent result.

Full character replacement needs to consider facial features, hairstyle, visible body proportions, clothing, posture, and interaction with instruments.

At the same time, the system should preserve the source performance's camera framing, choreography, and overall scene structure.

There is a tradeoff here. Allowing the model to reconstruct more of the person gives it additional freedom, which can also increase unintended changes to the original performance.

A promising direction is to separate identity and appearance references from motion and staging constraints, then evaluate each part independently.

I've found it useful to inspect results based on several dimensions instead of treating every completed generation as a success.

Identity accuracy: Is each uploaded person assigned to the correct performer?

Character consistency: Does the same person remain recognizable after a camera cut?

Temporal accuracy: Are the original shot boundaries and performance duration preserved?

Visual consistency: Do clothing, hairstyles, instruments, and character positions remain coherent?

Delivery reliability: Can the generated video actually be saved, played, and downloaded?

The last point is particularly important when turning an AI experiment into a web application.

Generation APIs may return asynchronous tasks, fail after submission, or provide temporary result URLs. A production system therefore needs persistent task states, reliable file storage, and recovery mechanisms for unsuccessful generations.

Based on these experiments, my next areas of investigation are explicit performer references, better control over original shot boundaries, and stronger validation of character consistency.

For tasks requiring precise preservation of camera cuts and performance timing, processing shots individually before assembling the final video may offer better control than generating the entire clip in one request.

However, that approach introduces additional complexity around motion continuity, editing boundaries, and audio synchronization.

The right solution depends on the video model's editing capabilities and the required level of fidelity.

Building multi-person AI video experiences has highlighted how difficult it can be to maintain consistent characters across a short sequence with multiple camera angles.

Even when individual frames look convincing, identity drift, unexpected transitions, and small timing differences can affect the final result.

My current focus with Tiny Desk AI is improving these aspects of the generation workflow and making the results more reliable.

I'm curious whether other developers working on reference-to-video generation have encountered similar identity consistency issues, especially when five or more characters appear across different shots.

What techniques have worked best for maintaining stable characters across camera cuts?