Nine Hours of Fine-Tuning, and Why the Product Still Uses References

The problem every generative video tool has

Put the same character in twelve shots and you want twelve shots of the same character. Models do not naturally do this. Ask for "a scruffy black-and-tan dog" twelve times and you get twelve different dogs, all plausible, none of them yours.

There are two ways out, and they sit at opposite ends of a trade-off.

Reference conditioning hands the model pictures. This is what Showspring ships: every shot carries reference images of the characters, props and locations it needs, plus start and end frames to pin the motion. Nothing is trained, so a new character works the moment someone uploads photos of it.

Fine-tuning changes the model instead. You train a small adapter, a LoRA, until the weights themselves know the character, and then you need no references at generation time at all.

I had only ever shipped the first one. So I built the second properly, on real characters, and compared them.

The run

Five dogs, two models each. Rusty and Oreo are mine and they carry the show, so they went first. Mimi, Esme and Molly are guest characters, which makes them the harder test: I did not pick them for being easy, and two of them look alike enough to be a genuine problem.

Two open-weights bases: a diffusion image model for stills, and a 33B omni-modal video model that generates picture and audio together. Rank 16 adapters on both, trained locally on a single RTX 5090. The card is the whole constraint: I render on the pruned int8 checkpoint at 19.5 GB and train on the NVFP4 one at 11.7, because int8 leaves nothing for optimiser state and activations, and the text embeddings get cached before training starts because the 14.6 GB text encoder cannot sit alongside the transformer either.

The three guest dogs took 9h 54m between them, measured end to end. Rusty and Oreo were trained earlier in the project and their figures are approximate, recovered from checkpoint timestamps rather than a stopwatch. The video runs came in at about half the wall clock of the image runs, which looks backwards for the bigger model: the image LoRAs trained on a multi-resolution bucket at 512, 768 and 1024, while the video LoRAs trained at 512 only.

Each finished adapter is about 290 MB. A LoRA is a small file, because you are shipping a patch to a model rather than a model.

What training actually looks like

The clearest way to see fine-tuning work is to hold the prompt still and watch the checkpoints. Same prompt, seed and settings, sampled every 250 steps.

For the first 750 steps you are looking at the base model's idea of a dog: a generic terrier, competently rendered and completely wrong. Somewhere around step 1000 the actual animal arrives, with the black-and-tan colouring, the upright pointed ears and the tan eyebrow markings. By 1500 it has her red collar too, which nobody put in the prompt. It picked that up from the photographs.

That last detail is the useful one. The adapter is not matching a description, it is reproducing a specific animal, including things I never thought to write down.

The same thing, in video

The video model got the same treatment. Below, each dog is generated twice from an identical prompt, seed and step count. The only variable is whether the trained adapter is attached, and neither side gets reference images.

The whole pipeline, with the voice on top

The adapter is one part of it. This clip runs the full production path: the video model generates picture and audio together with the character adapter attached, then his own synthesized voice is separated out and replaced with the canonical one through speech-to-speech, which keeps the timing the mouth was animated to. Sound on.

Putting a number on it

Eyeballing a video is not evidence, so here is a crude one. I embedded every curated training photo of each dog with CLIP, averaged them into a single vector for "what this dog looks like", and measured the cosine similarity of frames from both renders against it.

Every adapter moves toward its dog, 21% on average. The obvious objection is that this rewards matching the training distribution, which is why the renders are scenes that appear in no training set and why the number that matters is the gap between two otherwise identical renders rather than the absolute score. Treat the absolute values with suspicion, because CLIP scores any two photographs of any two dogs fairly high. The number worth reading is the gap between the two columns on an otherwise identical render.

The ordering is the interesting part. Oreo trained on 151 images and tops the table. Molly trained on 80, the second largest set, and sits at the bottom with a third of Mimi's gain on nearly four times Mimi's data. So the driver is not how many photographs you have.

It is how different they are from each other. Oreo's 151 came from years of ordinary photographs in different rooms, different light, different distances. Molly's 80 were dominated by 65 frames from a single afternoon at a dog park: one path, one light, one low camera angle. Mimi's 22 were varied and she nearly matched Rusty on 25. Diversity did the work, and volume mostly came along for the ride.

Where the time really goes

Most of the difficulty sat in deciding what to train on. Three things cost me a run each.

A quality filter can be a diversity filter with the sign flipped. Molly came with 300 candidate photos: 274 from one burst at a dog park, plus 26 varied everyday shots. I ranked that pool by sharpness to drop the motion-blurred frames, which looked correct in isolation. The burst was shot on a good camera, so it outranked the phone snaps, and the filter packed the set with one afternoon on one path while discarding the indoor shots, the coat and the portraits. Her final 80 ended up 65 dog-park frames to 15 everyday ones. The fix was to take a fair share from each source rather than rank the pool as a whole, and the +0.049 in the table is what the original filter cost.

Never train on a frame with two subjects in it. A white poodle wanders through a lot of those dog park photos. Guessing which animal is the subject is a coin flip you will eventually lose, so any frame with a second dog got dropped, 120 of Molly's. Fair-share sampling then took the survivors down to the 80 that trained. The rule earns its keep twice here, because Esme and Mimi are both black-and-tan, and one contaminated frame would teach one adapter the other dog's face.

Keep the class noun in the trigger word. An earlier run in this series stripped it, leaving a bare invented token with no anchor. The base model's priors filled the gap and by step 500 that adapter was generating a human man. Two words fixed it.

The comparison

That last cost bites hardest in production. Two character adapters in the same frame fight each other: at full strength the framing collapsed toward close-ups, because close-ups are what the training photos mostly are, and coat colour drifted on the earlier pair I trained for this show. Dropping both to about two thirds strength fixed it for pairs. Beyond that the answer is not a bigger card, it is coverage. Shoot the characters in separate takes and cut between them, the way a dialogue scene is actually shot, rather than trying to hold four identities in one frame.

Which one ships

Showspring keeps reference conditioning. Fine-tuning won on quality and lost on who can use it. Ten hours and a curated dataset per character is not something you hand to a customer who wants an episode this afternoon and does not own a 32 GB graphics card, and uploading eight photos is.

Fine-tuning is the right answer at the other end of the scale, where a character recurs across hundreds of shots and a few hours of preparation disappears into the run. I have not calculated the exact break-even for our cost per shot, and I would want that number before moving a production show onto adapters.

Building both is what made the call obvious. Reference conditioning looked like a compromise until I had a fine-tuned version to compare it against. Then fine-tuning looked like the upgrade, right up until I counted what it would take to hand it to a stranger with no GPU.

Models used: Qwen-Image for stills and MiniMax H3 for motion, both open weights. MiniMax H3 is used under written authorization from MiniMax to Overdigital LLC. Everything ran locally on my own hardware and nothing was redistributed. All timings measured on a single RTX 5090. Similarity figures are CLIP ViT-B/32 cosine against the mean embedding of each dog's curated training set, three frames per render. The characters belong to a show I produce; the training data is their owners' photographs, used with permission.