Animate, since renamed Pipeline Video, turns a website, a repository or a document into a narrated video. Before the render, an optional AI agent can edit the scenes, and that edit can go wrong. This post follows one offline test job in which it did.
The job had 18 seconds of narration and four timed scenes. The first validation check passed. A simulated AI agent then broke the timeline marker in ov00.html, the HTML file for the first scene. The second check caught it, and the job put the original version back before rendering once.
I ran this test to see the failure path. In a run where the agent’s edit passes, the fallback code never runs. This run shows what the code does when the agent’s edit fails validation.
The pipeline in seven steps
Making a video from a site takes at least seven steps: read the source, choose an angle, write the narration, turn it into speech, match each word to a time in the audio, build timed scenes, and check the rendered file.
Each step can fail on its own. A crawler can fetch the wrong page while the renderer works fine. A good script can get bad timing. A valid composition can still render a file with no audio.
If one model call does all of it, you cannot see which step broke. In this pipeline each step leaves a named value or file behind: SiteBrief, Plan, vo.mp3, a list of Beat objects, overlay HTML (one file per scene), CheckResult, and Probe. When a job fails, I look for the last of these that came out right. Figure 1 shows where each one is made.
Figure 1. The two halves of a job at commit b863a07, in the order pipeline/orchestrator.py runs them.
The user approves the script before anything costly runs
run_plan turns a website, repository, or document into a SiteBrief. From that it produces a Plan: the subject, the approved language, the requested length, and the script. The web flow stops there and sends the script back to the browser.
The user can fix a claim or change the angle. Only then do the costly steps start: voice synthesis, footage search, agent editing, and rendering. A wrong claim can be fixed before any of that is paid for.
At LumiereAI, we built Animate around this split. Its new name is Pipeline Video. Its public site accepts all three source types and shows the script in an editable box before rendering starts. Once the words are approved, run_video takes over (the lower half of Figure 1).
The voiceover sets the timing
The generated MP3 decides when everything happens. The code first reads its real duration.
Then it runs forced alignment. Given the audio and the known script, the aligner returns a timestamp for each word. The ElevenLabs Forced Alignment documentation describes exactly that contract: an audio file plus its text in, transcript timestamps out.
The word timings are then grouped into back-to-back Beat objects. A beat is one scene’s slot: it sets the scene’s start time and its length. At this commit, a new beat starts once the current one would run past 4.5 seconds and its last word ends a clause, with a full stop, a comma or a similar mark. Figure 2 shows the beats from the offline run described below.
Figure 2. Beats from the offline run. In this run a beat closed on each word that ended a sentence.
Alignment can fail. Then _beats_for falls back to splitting the measured audio duration across the script’s sentences, in proportion to their length in characters. That loses word-level precision but keeps three properties:
- the first beat starts at zero;
- each beat ends where the next begins;
- the last beat ends with the audio.
If any of those slipped, the video would show black gaps.
The scaffold and the agent
Two terms first. A composition is the HTML project the renderer turns into a video. The scaffold is a plain but complete composition that the code builds without the agent.
Composition.write builds the scaffold from the beats and the chosen footage. It writes one overlay HTML file per beat, the root index.html, the project configuration, and local font files. The voiceover audio already sits in the job’s workspace, and the root composition points to it.
The first check runs as soon as those files exist. If the scaffold is invalid, the job stops there, before any agent editing or rendering.
Only then does the optional agent get the scene files. It works in a throwaway workspace, and different agents can own different overlays without touching the shared index.html. The agent can change motion and styling. The timing is already fixed by the scaffold, which is ready to render.
A second check runs after the agent. The rule for what gets rendered:
- if the agent’s edit passes validation, render the edit;
- otherwise, render the scaffold;
- if the scaffold itself failed the first check, stop.
The smallest model of that rule uses two functions and one immutable data structure:
from dataclasses import dataclass
@dataclass(frozen=True)
class Composition:
name: str
has_timeline: bool
covers_voiceover: bool
def valid(composition: Composition) -> bool:
return composition.has_timeline and composition.covers_voiceover
def choose_for_render(scaffold: Composition, agent_edit: Composition) -> Composition:
if not valid(scaffold):
raise ValueError("invalid scaffold")
return agent_edit if valid(agent_edit) else scaffold
scaffold = Composition("scaffold", has_timeline=True, covers_voiceover=True)
agent_edit = Composition("agent-edit", has_timeline=False, covers_voiceover=True)
selected = choose_for_render(scaffold, agent_edit)
print(f"scaffold valid: {valid(scaffold)}")
print(f"agent edit valid: {valid(agent_edit)}")
print(f"selected for render: {selected.name}")
I ran that file with Python 3.12:
scaffold valid: True
agent edit valid: False
selected for render: scaffold
This toy only shows the rule. The test below imports the real pipeline.orchestrator.run_video and drives its failure branch with local fakes.
Test setup
The test uses commit b863a07e2ab64f5b47136345eda043fff49afab9. The script checks that the clone is on exactly that commit before importing run_video.
Every outside service is replaced by a local fake: speech synthesis, alignment, the language model that picks footage queries, footage download, the check, the agent, the renderer and the probe. The fake voiceover lasts 18 seconds. The fake aligner returns 18 placeholder words, one per second, with a full stop after the 5th, 10th, 15th and 18th word.
The fake agent edits ov00.html. It replaces a real timeline token, window.__timelines, with window.BROKEN_AGENT_TIMELINE. The fake second check reads the file and returns its prepared failure only if BROKEN_AGENT_TIMELINE is there. So the failure cannot happen unless the edit really landed.
The fake renderer writes one local file. The fake probe reads that file back and returns a Probe with a duration, width, height, and audio flag. The script copies those values from the call instead of typing them in, and records the commit only after confirming the clone is on it.
I also reran the repository’s test suite locally at the same commit. All 408 unit and integration tests passed in 12.94 seconds in the recorded run. Those tests replace real services with fakes too, so the time says nothing about speed or rendering.
Results
When run_video gets the failed CheckResult, it calls Composition.write again, which writes the scaffold over the agent’s edits. The script then rereads ov00.html and confirms the marker is gone. The job renders once and probes the file (Figure 3).
Figure 3. The broken marker changes which version reaches the renderer. The two checks, the scaffold rewrite and the probe still run in their recorded order.
The run produced four beats covering all 18 seconds, two validation calls, one confirmed scaffold restore, one render call, and one Probe reporting 1920 by 1080 pixels with audio. The job’s warnings record the reason: agent edits reverted, check failed: agent mutation removed a scene timeline. These are exact values from one offline run in which every outside service was a fake.
A timeout on its own takes a different path. If the files still pass the second check, the code keeps them and records the agent’s error as a warning. The scaffold is restored only when validation fails.
The probe checks the rendered file. At this commit, run_video adds a warning when the rendered file’s width or height differs from the composition’s, when it has no audio track, or when its duration is more than one second away from the voiceover’s.
Progress messages should name the step that is running
Agent editing and rendering take longer than a normal web request. So the API returns a job ID right away, and the browser follows progress through server-sent events.
Server-sent events fit because the data flows one way, from server to page. The MDN guide describes EventSource as a one-way connection from server to page. The run emitted these stages in order: recording the voiceover, timing the scenes, sourcing footage, building the composition, directing the scenes, rendering, verifying, and done.
If the page stalls on “timing the scenes”, I look at alignment. If it stalls on “directing the scenes”, I inspect the workspace and the agent’s events.
What this experiment does not show
No paid API was called. The MP3, footage clips, and MP4 are placeholder bytes. The probe is strict and its output was captured from the call, but it is still a fake. So this run says nothing about visual quality, voice quality, speed in production, cost, throughput, or customer adoption.
It shows one narrower thing: when an edited file makes the second check fail, the code at that commit rewrites the valid scaffold, renders once, and runs its check of the finished file.
A real end-to-end test would need fixed provider versions, real rendering hardware, failed jobs kept for inspection, frame and audio checks, and a quality rubric written before anyone watches the outputs.
The checks I would keep before adding another model
- Send the complete script back for approval before synthesis or rendering.
- Take scene timing from the generated narration.
- Build a complete composition with no agent output.
- Run the post-agent check on the files the agent changed.
- Restore the scaffold only when those files fail validation.
- Probe the rendered file for duration, dimensions, and audio.
- Emit stage names that point to the part of the system responsible.
If I add another AI step, it has to hand the next stage an output that a check can reject. If I cannot name that output and its check, the step is not ready for the pipeline.