An 18 second offline job produced four timed scenes. Its first composition check passed. A simulated agent then broke the timeline marker in ov00.html, the second check caught it, and the job restored the scaffold before rendering once.
That is the failure path I wanted to see. A successful demo would only tell me that the happy path works. This run tells me where creative output stops and delivery control begins.
Reproduce the selection rule in one file
The smallest model uses two functions and one immutable data structure:
from dataclasses import dataclass
@dataclass(frozen=True)
class Composition:
name: str
has_timeline: bool
covers_voiceover: bool
def valid(composition: Composition) -> bool:
return composition.has_timeline and composition.covers_voiceover
def choose_for_render(scaffold: Composition, agent_edit: Composition) -> Composition:
if not valid(scaffold):
raise ValueError("invalid scaffold")
return agent_edit if valid(agent_edit) else scaffold
scaffold = Composition("scaffold", has_timeline=True, covers_voiceover=True)
agent_edit = Composition("agent-edit", has_timeline=False, covers_voiceover=True)
selected = choose_for_render(scaffold, agent_edit)
print(f"scaffold valid: {valid(scaffold)}")
print(f"agent edit valid: {valid(agent_edit)}")
print(f"selected for render: {selected.name}")I ran that file with Python 3.12:
scaffold valid: True
agent edit valid: False
selected for render: scaffoldThis example only exposes the selection rule. The larger experiment later in the article imports the real pipeline.orchestrator.run_video and drives its failure branch with local fakes.
One prompt hides too many failure modes
“Read this site and make a good video” sounds like one task. The implementation has to solve at least seven.
It reads the source, chooses an angle, writes spoken copy, synthesizes narration, aligns words to time, builds timed scenes, and inspects a rendered file. A crawler can return the wrong page while the renderer remains healthy. A good script can be paired with bad timing. A valid composition can still produce a file without audio.
A single model response gives those failures nowhere to land. In this pipeline they land in named values and files: SiteBrief, Plan, vo.mp3, a list of Beat objects, overlay HTML, CheckResult, and Probe.
When a job fails, I want the last valid name.
Approval splits planning from spending
run_plan resolves a website, repository, or document into a SiteBrief. It then produces a Plan containing the subject, approved language, requested duration, and script. The HTTP flow stops there and returns the words to the browser.
The user can edit a claim or replace the angle before voice synthesis, footage search, agent editing, and rendering begin. That boundary also keeps a semantic problem out of the expensive half of the job.
At LumiereAI, we built Animate around this split. Its public interface accepts all three source types and puts the script in an editable field before the render starts. Once the words are approved, run_video takes over.
The voiceover owns the clock
The generated MP3 is the first timing authority. The code reads its actual duration, then passes the audio and known transcript to forced alignment. Word timestamps are grouped into contiguous Beat objects, and those beats become scene start times and durations.
The ElevenLabs Forced Alignment documentation describes that input contract directly: an audio file plus its text yields transcript timestamps.
Alignment can fail. The fallback in _beats_for distributes the measured audio duration across script sentences by character count. That loses word-level precision while preserving the part that would cause black gaps if it drifted: the first beat begins at zero, adjacent beats touch, and the final beat reaches the end of the audio.
Composition.write creates the floor
Composition.write receives the beats and footage choices. It writes one overlay HTML file per beat, the root index.html, project configuration, and local font files. The synthesized audio already exists under the job workspace and is referenced by the root composition.
The first call to check happens immediately after those files are written. An invalid scaffold stops before agent editing or rendering.
Only then does the optional authoring pass receive scene files. Its workspace is disposable, and different agents can own different overlays without editing the shared index.html. The pass can change motion and treatment. The scaffold has already established complete timing and a renderable project.
I also reran the pinned repository suite locally. All 408 unit and integration tests passed in 12.94 seconds in the recorded run. Those tests use injected fakes, so the number is a development check rather than a speed or rendering benchmark.
The second check has evidence to inspect
The offline experiment uses commit b863a07e2ab64f5b47136345eda043fff49afab9. The reproduction script verifies that the clone is on that exact commit before importing run_video.
Its fake author edits ov00.html by replacing a real timeline token with BROKEN_AGENT_TIMELINE. The second fake check reads the file and refuses to return a canned failure unless that marker is present. When run_video receives the failed CheckResult, it calls Composition.write again. The reproduction then reads ov00.html once more and confirms that the broken marker is gone.
A timeout by itself follows a different path. If the files still pass the post-agent check, the orchestrator keeps them and records the agent error as a warning. Scaffold restoration is tied to failed composition validation.
The render fake writes one local artifact. The probe fake reads it, constructs a Probe, and the reproduction serializes the same observed duration, width, height, and audio flag from that invocation. No probe field is hardcoded; the source commit is recorded only after Git HEAD is verified.
Figure 1. The broken marker changes which composition reaches the renderer. The two checks, scaffold rewrite, and media probe still execute in their recorded order.
The run produced four beat windows covering all 18 seconds, two validation calls, one verified scaffold restoration, one render call, and one Probe with 1920 by 1080 pixels plus audio. These are exact fixture observations from one offline scenario.
Progress should name the active boundary
Agent editing and rendering outlive a normal request. The API returns a job identifier immediately and the browser listens for progress through server-sent events.
That transport fits the direction of the data. The MDN guide defines EventSource as a one-way connection from server to page. The observed run emitted these stages in order: recording the voiceover, timing the scenes, sourcing footage, building the composition, directing the scenes, rendering, verifying, and done.
If the page stalls on “timing the scenes,” I look at alignment. If it stalls on “directing the scenes,” I inspect the workspace and agent events. The last emitted stage is useful because it names the subsystem instead of displaying a generic spinner.
What the experiment leaves open
No paid API was called. The MP3, footage clips, and MP4 are fixture bytes. The probe is strict and state-captured, but it remains an injected boundary. This run therefore says nothing about visual quality, voice quality, production latency, cost, throughput, or customer adoption.
It proves one narrower statement: after a file mutation makes the second composition check fail, the pinned orchestrator rewrites the valid scaffold, renders once, and runs its media verification path.
A real end-to-end evaluation needs frozen provider versions, real rendering hardware, retained failed jobs, frame and audio inspection, and a quality rubric written before anyone watches the outputs.
The checks I would keep before adding another model
- Return the complete script for approval before synthesis or rendering.
- Derive scene timing from the rendered narration.
- Build a complete composition without creative agent output.
- Let the post-agent check inspect the files the agent changed.
- Restore the scaffold only when those files fail validation.
- Probe the rendered file for duration, dimensions, and audio.
- Emit stage names that point to the responsible subsystem.
If I add another model-generated step, it has to hand the next stage something concrete enough to reject. If I cannot name that artifact and its check, the step is not ready for the pipeline.