The Check Passed. The Film Was Broken.

Nobody films anything. I write a script, a voice reads it, software goes looking for an archival photograph for each line, a video model animates some of them, an on-camera presenter is generated shot by shot, and the pieces are cut into something about fifteen minutes long with chapters, a score, captions and an end card. Eleven films have come out of that in about three weeks, on two GPUs in my house. Almost none of it needs me while it runs, which is the part that is new.

The interesting part is not that it can be done. It is what has to be true for the output to be worth watching, and my answer to that has changed twice this year.

The models were never the hard part

In September I published a fifteen-minute documentary in which 45 of the 62 on-camera shots were out of sync, the worst of them by more than a second. I had a test for exactly that failure. It passed.

It was not a flaky test and it was not a bad measurement. The test did precisely what it was written to do, on precisely the file it was written to look at, and that file was not the film.

Some of these films have an on-camera presenter, a generated person who speaks to the lens. The presenter is rendered one line at a time, and each of those clips is a take: a few seconds of video, generated to match one specific piece of audio. A film has dozens of takes, and one take becomes one shot in the finished film. Once the timing is settled they are cut together, and the file a viewer watches is that cut.

The lip-sync test measured every take against the audio it had been told to copy, and by that measure the clips were fine. It never opened the assembled film. The assembler then placed each shot's picture at the moment the first spoken word begins, while each take had been lined up with the start of its whole audio file, and that file opens on a little silence. So in the assembled film the picture sat late against the voice. Always late, never early, by the silence at the front of its own audio. The median lateness was 0.20 seconds and the worst was 1.16.

Out of sync here means past 0.15 seconds, which is a constant called MAX_OFFSET in the check. I come back later to why that number gets to live in the code and another one of mine does not.

Months earlier, after a different film had gone wrong in a related way, I had written a sentence in a notes file: measure the delivered film, not the takes. The run does not open that file. I never connected the sentence to anything that executes, so it never ran against the cut that shipped.

That is the whole lesson, and it has nothing to do with model quality. Every model in that pipeline did its job. The film was broken anyway, because the loop that was supposed to catch it was measuring the wrong artifact.

The feedback loop is the product

Full automation is not a generation problem. It is a verification problem. The generator is allowed to be wrong as often as it likes, provided something downstream reliably notices, and the entire difference between a pipeline that produces watchable films and one that produces plausible garbage sits in that sentence.

This is not just my experience. A benchmark published on the first of September evaluated five production stages, from script through storyboard and keyframe to finished drama, across 5,785 items and 17,488 human scores, and its central finding was that upstream defects cascade through the pipeline and final quality is not governed by video generation alone. Improving the video model does not fix a bad storyboard. A pipeline that only inspects the final render will not find where the defect entered.

So after the cats film I stopped starting a project by copying the previous film's configuration, and I built the loop properly. Only one film has been made since. The other seven in the numbers below are back catalogue, re-run through the checks after the fact, which is how I know what the old ones were hiding.

A check aimed at one file

There are 62 checks. That 62 is not the 62 on-camera shots from the opening. Each check record carries the defect it looks for, the file it measured, the number it got, and the incident that caused me to write it.

Here is the record the cats check writes now, on the repaired cut:

check     delivered-lipsync
artefact  delivered_film          (cats_OUTRO.mp4)
severity  blocker
result    pass
detail    every piece within 0.15s; median |offset| 0.02s
incident  cats documentary, 11 September

The artefact line is the point. In September that field would have said the takes, and the row would still have said pass.

Underneath those checks sit 463 unit tests that run in 40 seconds and test the checks. A check I have not tested is a more confident version of the note I started with.

The table is six of the checks, each row naming where I learned it.

The check	What it looks for	Learned from
picture-face-crop	A photograph cropped through somebody's head	Year 1816
picture-period	A modern object in a film set 200 years ago	Year 1816
scene-speaker-visible	Someone off screen during their own line	An earlier film
picture-repeat	The same photograph coming back seconds later	Two films
score-instrumental	A singer in music meant to be instrumental	Antikythera
direction-spoken	The narrator reading a stage direction out loud	Cats

On the last of those, the script had a note in square brackets describing how a line should be delivered. The narrator read the brackets aloud, in character, and that reading went into a finished cut.

Detecting a cropped face is ordinary measurement. A modern object in a film set two hundred years ago is a question for a vision model, so that check builds a contact sheet and asks one. The answer is filed as a warning. I do not let that warning stop a release on its own.

Keep the measurement, not the verdict

Every measurement goes into one database. The 59,637 above is the row count as I write this, one row per measurement, and it was still going up while I typed the number.

When the narrator finishes a line, the shot is often still running. The video model has to keep going through that gap, and leftover silence is how long the gap is. I had made every one of these films before I looked at the distribution.

I could not have told you the median was 0.77 seconds, or that 22 shots leave the model more than three seconds. The first chart I made of this was wrong, because I counted the wrong unit. The database holds one row per measurement, not one per shot, so a shot I had rendered thirty times contributed thirty rows. I plotted the rows. The shots I had re-rendered most were counted most, and the tail came out fatter than the films are.

That is the difference between the two numbers I keep mentioning. 0.15 is a constant in a check, so it fires on a named file and writes what it measured. 0.45, the jump between shots I decided one evening was visible, is a belief, and it has been steering cuts since the first film without ever having to show where it came from. The rows I would need to settle it did not exist in June. They exist now and I have not gone back and done it.

What the checks still miss

On a Sunday in September I watched two cuts and the furniture was wrong in both. On the last line of one film the presenter was standing in the desk. In another, on a line about selling a computer in a department store, the presenter was sitting inside the workbench, the bench top showing at the left and right edges with his torso coming down through it to the keyboard.

Nothing I had written measured where a body sits in a room. Those shots had been checked for the words, for the mouth, and for whether the face was the right face. The checks passed. I was looking at the cut while the furniture went through the body.

I then tried to write a check so it would not depend on me happening to be watching. It compared each frame with a photograph of the empty set, on the assumption that the presenter is pasted on top of it. The video model does not work that way. It repaints the whole frame, lighting and grain included, so nearly every pixel differs. On one film 32 of 37 takes came back marked broken, including close-ups of a face with no desk in the shot, and it also missed one take I had already seen by eye. I switched it off the same day.

Some time after the rebuild, going through a cut by hand, I found fourteen bad shots across two films, ten of them in the cats film and four in an older one. When a take is unusable, a repair step swaps it for a held still of the speaker's face, and that swap also removes the shot from the list the check reads. The check then looks at a shorter list and comes back green. I found the first one by eye, 39 seconds in.

This is worth saying plainly because it is the honest state of the art: every defect in this post was found by a human, and the one time I tried to automate the catch, the automation was worse than me.

How fast this is actually moving

The reason this is worth writing down now is that the ground moved underneath it while I was working.

Here is what I installed on those two machines between the first of August and today, taken from the caches rather than memory.

Date	What landed	Layer
02 Aug	HunyuanVideo-Foley-XL	foley
02 Aug	Chatterbox, Qwen-Image	speech, image
11 Aug	MiniMax H3	video and on-camera
15 Aug	MiniMax Music 3	score
07 Sep	HTDemucs	source separation
08 Sep	ACE-Step 1.5, Breeze TTS 2, Step-Audio-EditX	score, speech
16 Sep	faster-whisper base, small	transcription
19 Sep	Qwen3-TTS VoiceDesign	speech
22 Sep	Qwen-Image 2512	image
23 Sep	faster-whisper medium.en, large-v3	transcription

Most of that stack turned over in seven weeks. Speech went through three generations, transcription through four sizes with the last two this morning, and the score and image models each changed once. Foley and the video model are the exceptions: they arrived at the start of the window and have not been replaced yet, and the on-camera presenter only exists at all because a 33 billion parameter video model with native audio got open weights at the start of August, which is the 11 Aug row above.

That is the thing I would ask a sceptical reader to sit with. Not any single release, but the replacement rate. A pipeline built around one specific model is a pipeline you will rewrite every six weeks. A pipeline built around a loop that measures the delivered artifact does not care which model produced it, and that is the only reason any of this survived August.

Two caveats I would not skip. The open weights got easier to obtain and harder to use commercially in the same seven weeks, and more than one of the models above ships under a research or non-commercial licence that a hobby project can ignore and a business cannot. And the hard problems did not move much: character consistency across shots, camera you can actually specify, and coherence past about seven seconds are all still being solved outside the model rather than inside it, by reference banks and geometry and exactly the kind of external scaffolding this post is about.

The two films

Commodore is about how Jack Tramiel lost the company he built. It is the first film made after the rebuild, and its narration was rendered on the GPUs in my house.

The cats film is the one from the opening, in the repaired cut that is public now.

I never wrote down how many defects reached a published film before September, so there is no before and after to show, and I am not going to make one up. The claim I can make is smaller. Every check now names the file it opens and writes its number into the log, so a green result says which cut was measured. The test that passed while 45 shots sat out of sync did not have to say that.

One thing I have changed my mind about

The other system I built, a production tool for episodic shows, is architected the way I used to think this work should be architected. Specialised models, each given the job it is best at, coordinated by an orchestrator. Counted this morning, that codebase carries LLM callsites in 64 files: 261 to one model family, 111 to a second, 52 to a third, plus 95 through a local bridge and 11 to a local model.

It works. What I notice now is where it goes wrong. Every callsite gets the slice of context its step was designed to need, so the model writing a line does not know what the picture two shots later will be, and the model choosing a photograph does not know the cadence of the narration it will sit under. The orchestrator knows the order of operations, not the film. The continuity failures in that system are not model failures. They are the seams between the calls, and I spent months writing code to carry context across seams I had created myself.

The documentary pipeline went the other way, less by design than by circumstance. One model has the script, the beat timings, the photographs, the check results and the failure history in front of it when it decides. There are fewer seams because there are fewer handoffs, and when something is wrong it tends to be wrong somewhere the cause is also visible.

I want to be careful about how much weight that carries, because it is not an experiment. The two systems do different jobs, for different audiences, at different ages, and I have cared about them unevenly. Nothing here isolates architecture from everything else that differs. It is an observation from building both, not a result.

There is at least one study pointing the same way. Dennis and colleagues at Melbourne, at the end of April, compared putting a whole procedure into one model's context against running the same model behind a graph orchestrator, across travel booking, technical support and insurance claims. The single-context version failed less in all three: 11.5, 0.5 and 5 percent against 24, 9 and 17. Same frontier model on both sides, so the orchestration was the thing making it worse.

Their domains are procedural and mine is not, which is the honest limit of borrowing it. Nobody has run that comparison on anything as soft as whether a film is any good, and I am not sure what the measurement would even be.

And the honest caveat is the one this post has already made for me. A single model holding everything is only safe to the degree the loop around it is real, and the furniture section above is what that looks like when it is not: the one time I tried to automate a check for a defect I could see with my own eyes, the automation was worse than me and I turned it off the same day. So I hold this as a direction rather than a conclusion. Fewer seams, more measurement, and a great deal still found by watching.

What I did not expect when I started is how much of this turns out to be ordinary engineering. Name the artifact. Measure the thing you ship. Keep the distribution, not the verdict. Do not let a number steer the work until it can say where it came from. None of that is specific to film, and none of it required a better model.

The models will keep arriving, faster than anyone can integrate them, and most of what makes them useful for storytelling is not in the weights. It is in whether the loop around them is honest. I find that genuinely encouraging, because a loop is something you can build.

I am starting a new full-time role, so this is the last of these for a while. The films carry on the way they always have, mostly on their own, on the two GPUs in the house. The checks carry on too, and they still cannot see a man standing inside a desk.