Building an AI rap video workflow with scene previews and recoverable jobs

A ten-second rap video sounds like a small job: take a portrait, generate a performance, return an MP4.

A request can time out after the provider has already accepted the job. The video might then finish after the user closes the tab. And before any of that, there is a simpler problem: the generated scene might have the wrong face.

I'm building RapVideo, a web app that turns one or two portraits into short rap performances. Its main workflow has two generation steps, with a scene review between them. The application uses React and TanStack Start, runs on Cloudflare Workers, and keeps task state in D1. Generated media goes into R2.

The prompts are only one part of the implementation. Much of the code deals with what happens after a request leaves the server. I've shortened the examples below to keep that logic readable.

portrait(s) + scene choices ↓ scene image generation ↓ user checks the scene ↓ video generation with audio ↓ stored MP4 + task history Enter fullscreen mode Exit fullscreen mode I want the user to inspect the cast and composition before committing to animation. A still image makes that inspection easier. Are both people present? Are their faces distinct? Is the microphone in an awkward position? Those are useful questions to answer before asking a video model to add movement.

The review is also a spending decision. In the current workflow, the scene uses 10 credits and the video uses 90. Generating another scene is a separate paid attempt. I don't describe the first step as a free preview, because it isn't one.

An approved scene doesn't guarantee stable faces throughout the video. It gives the user a checkpoint, and it gives the video request a specific starting frame. The app retains that scene while animation is pending.

For the main rap workflow, the choices are deliberately narrow: solo or duo, a studio/street/stage setting, and a friends/birthday/team theme. Solo requires one reference photo; duo requires two. This particular workflow doesn't expose arbitrary lyrics or uploaded audio.

That makes the request easier to reason about. The browser sends choices and reference identifiers. A versioned server template decides the provider route, prompt, input keys, and credit charge.

const request = { templateId: " rap ", templateVersion: 1, step: " scene ", settings: { mode: " duo ", stage: " studio ", topic: " friends ", prompt: "", // This template doesn't accept freeform notes. }, references: [ firstPortraitId, secondPortraitId ], }; Enter fullscreen mode Exit fullscreen mode The server validates each choice against the template. It also checks ownership of the referenced assets and signs them for provider access. In this path, uploaded references must be JPEG, PNG, or WebP files under 10 MB.

I prefer keeping these decisions on the server. An edited browser request should never be able to choose a different model or change the number of credits charged. A disabled dropdown is useful UI, but it doesn't enforce either rule.

Template versions also stay with the task. When investigating an older result, I want to know which configuration produced it, even if I've changed the default since then.

Here's a fragment from the duo scene prompt:

Exactly TWO adult performers. Reference image 1 stands on the LEFT and reference image 2 on the RIGHT. Preserve their distinct faces, hairstyles and outfits; do not blend their identities. Enter fullscreen mode Exit fullscreen mode The studio setting adds a recording booth, dark acoustic panels, a suspended microphone, and warm amber lighting. Another part specifies the framing:

Face the camera, relaxed confident posture, hands visible, knees-up framing with comfortable space around the head. Enter fullscreen mode Exit fullscreen mode I care more about these instructions than a long string of aesthetic adjectives. They describe things a person can check in the resulting image. Left and right are particularly useful because the video prompt can refer to those same positions later.

The scene prompt also asks for no extra people, text, or logos. Those are requests to a generative model, so I still expect the output to need review. Writing “exactly two” doesn't turn a model into a cast-count validator.

That first image is also where I start debugging a bad result. If the scene already has a blended face, changing the video prompt is a poor place to start. Fix the reference or scene first, then inspect the next result.

The second step receives the generated scene as its starting image. The current video template uses Kling 2.6 through Replicate, with a ten-second duration and generated audio enabled. Its provider input includes:

const input = { start_image: signedSceneUrl, duration: 10, generate_audio: true, prompt: compiledPerformancePrompt, }; Enter fullscreen mode Exit fullscreen mode The performance prompt asks for a steady portrait shot, small rhythmic gestures, and an original hip-hop beat. For a duo, it assigns the two short lines to different positions:

From seconds 1 to 4 the LEFT performer raps: "Good friends together, we bring our own flow." From seconds 5 to 8 the RIGHT performer replies: "Side by side shining, wherever we go." The silent performer nods along. Enter fullscreen mode Exit fullscreen mode It also asks the model to preserve the number of people and avoid cuts. I want limited movement here because the task already includes faces, hands, vocals, and timing. There isn't much room in ten seconds for an elaborate camera sequence.

Those timestamps express an intention. They don't give the application an audio timeline with frame-accurate edits. Words, pronunciation, turn-taking, and lip movement can vary. The prompt requests original audio and excludes imitation of a named artist; the workflow doesn't clone a supplied voice.

For evaluation, I'd watch the faces through the entire clip and listen to the actual words. A convincing first frame is an incomplete test of a video, especially when two people are supposed to take turns.

The submission path has an awkward boundary:

app reserves a task app sends a provider request provider accepts it connection drops before the app receives the response Enter fullscreen mode Exit fullscreen mode At that point, “try again” might create another paid provider job. Treating the timeout as a definite failure could also restore credits while the original job continues running.

The implementation distinguishes an explicit submission rejection from an unconfirmed outcome. It creates the local task before making the provider call. If acceptance is uncertain, it keeps the task pending for reconciliation instead of immediately submitting again.

The purchase flow derives the task identity from the user, tool, flow, purchase reference, step, and attempt. Repeating the same attempt returns the existing task. Asking for a new scene is a new attempt, which is a different action from retrying an interrupted HTTP request.

There is still a limit here. If the app never obtains a provider prediction ID, it may be unable to identify that particular provider job later. A local idempotency key doesn't magically make an external API exactly-once. Keeping an explicit uncertain state is more honest than pretending the network gave a definitive answer.

Before submission, the application reserves the task and debits credits in one atomic database operation. That operation also checks the user's generation concurrency limit.

begin atomic operation debit credits insert pending task enforce concurrency limit commit submit the reserved task to the provider Enter fullscreen mode Exit fullscreen mode The actual implementation includes guards around credit consumption and concurrent reservations. The aim is to avoid a debit with no task to recover, or a task that starts without funding.

The external provider call happens after that database operation. It can't participate in the same transaction, which is why the pending task and uncertain-submission handling matter.

Failure settlement needs similar care. A refund restores the balance associated with the original consumption and records an audit event. It must remain safe when another poll or recovery run observes the same failure. A second notification should find the task settled, rather than manufacture another credit grant.

The browser is useful for showing progress, but the job must survive without it. The code has a shared reconciliation function used by polling, history refresh, the webhook handler, and scheduled recovery.

Cloudflare runs the scheduled recovery every five minutes. For a pending task, reconciliation examines saved outputs, queries known provider predictions, persists available media, and settles the task when it has enough information.

Several callers can arrive at once. A short database lease limits concurrent reconciliation of the same task. The lease lasts two minutes, and the outbound work has a shorter, 90-second budget. Settlement checks the lease token so an older worker can't finish later and overwrite a newer recovery decision.

The webhook handler verifies the signature and timestamp, checks the local prediction mapping, and queries the provider's current state. It doesn't accept an arbitrary output URL from the incoming payload and download it blindly.

The recovery policy also has a one-hour deadline. When an unresolved provider job reaches that boundary, cancellation can race completion. The code checks the resulting provider state before deciding how to settle it. Otherwise, a video that finishes at the wrong instant could be treated as a failure.

A provider saying “succeeded” still leaves the application with work to do. The output has to reach storage and remain discoverable from the local task.

I use stable storage keys based on the task ID, prediction ID, and output index. Each object also carries metadata for those identifiers. Recovery checks for an existing object before downloading the same output again.

This handles a particularly annoying interruption: the R2 upload succeeds, then the database update fails. On a later run, recovery can find the uploaded object and attach it to the task. It doesn't need to assume that the missing database result means the media disappeared.

A failed storage check is also different from an empty result. If R2 is temporarily unavailable, the recovery function leaves the operation retryable. It doesn't turn that outage into evidence that no output exists.

For downloads, the code restricts accepted supplier hosts, rejects redirects, and checks media type and size. A remote URL is still untrusted input, even when it appears in a generation response.

The repository includes recovery tests for repeated callbacks, storage failures, and completion racing cancellation. There is also a test where a video completes after sixteen minutes with no browser present, plus a case that recovers an uploaded object after the provider output is no longer available.

These controlled tests describe the behavior I expect; they don't give me a production reliability percentage. I can change a refund branch and check whether it now restores credits twice, or change storage handling and see whether an already-uploaded result gets lost.

Visual quality needs a separate check. State-machine tests can't tell me whether a face drifts or whether the second performer delivers the right line. I need both kinds of review before calling the workflow useful.

I still like having the scene checkpoint. It gives the person using the app a place to make a decision, and it gives the backend two understandable jobs to track. When a clip goes wrong, I want to be able to tell whether I should change the scene, adjust the performance prompt, or investigate a delivery failure.

If you'd like to see how that looks in a working interface, I'm building it at RapVideo. Start with the scene, check the faces, and listen to the whole clip before deciding whether to keep it.