AI Movie Production Process

A complete, step-by-step pipeline for transforming short stories into fully produced AI-generated films using open source tools.

---

Quick Start

# Clone this template to start a new movie project
git clone https://spacecruft.org/movies/process my-movie-project
cd my-movie-project

# Initialize Git LFS (one-time per machine)
git lfs install

# Copy and configure environment
cp .env.example .env
nano .env   # Set API URLs, model paths, GPU config

# Start at step 000
cat 000-project-setup/README.md

Then work through each numbered directory in order. Each directory has a README.md explaining exactly what to do, what inputs you need, and what outputs to produce.

---

What's Inside

78 step directories (count as of 2026-09-02; recompute with ls -d [0-9][0-9][0-9]-*/ | wc -l — the step table in docs/OVERVIEW.md must equal that listing) organized into the phases below, with 6 GPU swap points:

Phase	Steps	Description
Pre-Production — Story & Script	000–081	Source story; LLM extracts characters, locations, plot structure, themes, screenplay, props, storyboard, style guide; 081 merges the character-relationship pair files (LLM-free aggregator)
Pre-Production — Visual Prompts	082, 084	LLM extracts visual descriptions for characters (082) and locations (084)
🔄 GPU Swap 1: Unload LLM → Load ComfyUI
Pre-Production — Visual Design	083–098	083 cast differentiation (probe render + AuraFace measure), then design images via ComfyUI for characters (090), locations (092–094, PyraCanny ControlNet), props (098)
🔄 GPU Swap 2: Unload ComfyUI → Load Qwen3-VL
Pre-Production — Dataset Curation	100–104	VLM rate (Qwen3-VL-Reranker + DreamSim/Embedding), curate, and caption training images
🔄 GPU Swap 3: Unload Qwen3-VL → Load AI Toolkit
Pre-Production — LoRA Training	110–114	Train Qwen-Image LoRAs for characters, locations, props (AI Toolkit)
Pre-Production — Identity Calibration	115–116	AuraFace + DINOv2-giant + SigLIP2 centroids and thresholds from the design images (local GPU)
🔄 GPU Swap 4: Unload AI Toolkit → Load ComfyUI
Pre-Production — LoRA Validation & Recalibration	120–124	120/122/124 checkpoint + strength selection (identity folded into the primary score; fatal quality gates); 121 LoRA drift gate + render-relative identity calibration read by every identity-aware runtime step
🔄 GPU Swap 5: Unload ComfyUI → Load LLM
Production — Shot Planning	160	Shot manifest (storyboard entities → LoRAs), LLM-authored per-shot prompts, per-scene mood boards
🔄 GPU Swap 6: Unload LLM → Load ComfyUI
Production — Keyframe Generation	170–180	Composition scouts (location LoRA only), GDINO/SAM2 segmentation, plate generation/selection, layered inpainting (character → prop), harmonization, face refinement, keyframe selection, keyframe QA, outpainting
Speech (before the storyboard)	066–069, 071	Qwen3-TTS voice design; whole-line synthesis with a Whisper intelligibility gate; MOS ranking, forced alignment and loudness normalisation; per-shot cuts at the storyboard's sentence boundaries (the storyboard plans on measured speech; LongCat consumes the cuts)
Production — Video Generation	220–270	Deterministic per-shot strategy (220), VLM prompt grounding (225), LongCat-Video-B200 clips with lip sync (230), verdicts (240), reshoots (245), final selection (250), upscale (260), finalize (270)
Production — Audio: Sound & Music	300–350	FreeSound CC0 retrieval, MOSS-SoundEffect SFX generation, Qwen3-Omni SFX selection, ACE-Step 1.5 XL Turbo music composition
Post-Production — Assembly & Finishing	360–430	Rough cut, transitions, audio mix, grade, titles, credits, subtitles, export
Review & Delivery	440–480	Quality review, issue log, revisions, feedback, ship

Key Tools & Infrastructure

• LLM — OpenAI-compatible API for all text analysis, screenplay, storyboard, prompt extraction, and shot planning
• ComfyUI — Image generation with multi-GPU parallel dispatch (COMFYUI_API_URLS), PyraCanny ControlNet for location consistency
• AI Toolkit — LoRA training for Qwen-Image models (characters, locations, props) with parallel GPU training
• Qwen3-VL — Dataset curation and LoRA validation (Reranker-8B for text relevance, Embedding-8B for cluster consistency, 30B-A3B-Instruct for captioning)
• DreamSim — Perceptual similarity scoring (CLIP + OpenCLIP + DINO ViT-B/16 ensemble) for curation, hero selection, and LoRA validation
• Grounding DINO + SAM2.1 — Text-guided object detection and pixel-precise segmentation for keyframe entity masking
• LongCat-Video-B200 — Video generation with audio-driven lip sync and multi-character support; a fork pinned by LONGCAT_FORK_COMMIT in 230-video-clip-generation/download.sh (step 245 clones the same commit and asserts equality)
• Qwen3-TTS — Voice design (VoiceDesign model) and whole-line speech synthesis (Base model, ICL cloning); Whisper-large-v3 gates intelligibility, wav2vec2 CTC alignment places the cuts
• FreeSound — CC0 sound effects retrieval via API with LLM-generated search queries
• MOSS-SoundEffect — Text-to-audio SFX generation (8B, duration-controlled)
• Qwen3-Omni — Audio understanding model (30B-A3B MoE) for SFX candidate evaluation and selection
• ACE-Step 1.5 XL Turbo — AI music composition (4B DiT + 5Hz LM, theme, scene scores, stingers)
• Configuration — All settings via root .env.example (API URLs, model paths, GPU IDs, training hyperparameters); required variables have no in-code default and a step exits non-zero when one is unset
• Rules — the .kilo/rules/ directory (see Rules)

Models

• ACE-Step 1.5 XL Turbo DiT — AI music composition diffusion transformer
• ACE-Step 5Hz LM — AI music composition language model
• AuraFace-v1 — face identity embedding model used for character identity calibration
• CLIP ViT-B/16 — vision-language backbone used in DreamSim ensemble
• DINO ViT-B/16 — self-supervised vision backbone used in DreamSim ensemble
• DINOv2-giant — self-supervised vision backbone for body identity embeddings in character/prop calibration
• DreamSim — perceptual similarity ensemble for image-pair distance scoring
• GLM 5.1 — text LLM for story analysis, screenplay, storyboard, prompt extraction, shot planning
• Grounding DINO — text-guided object detection for entity segmentation
• LibreHPS-4B v1.1 — aesthetic / human-preference scorer
• LongCat-Video — image-to-video generation model
• LongCat-Video-Avatar-1.5 — avatar video generation with audio-driven lip sync
• MOSS-SoundEffect — text-to-audio sound effect generation
• OpenCLIP ViT-B/16 — vision-language backbone used in DreamSim ensemble
• Qwen-2.5-VL-7B — CLIP text encoder for Qwen-Image
• Qwen-Image — image generation diffusion model
• Qwen-Image ControlNet-Union — ControlNet for location design + character/prop inpainting
• Qwen-Image-Edit — image editing diffusion model for outpainting
• Qwen-Image VAE — variational autoencoder for Qwen-Image
• Qwen3-Omni — audio understanding model for SFX evaluation and selection
• Qwen3-TTS Base — text-to-speech base model for dialogue and narration with ICL voice cloning
• Qwen3-TTS VoiceDesign — text-to-speech voice design model for voice profile creation
• Qwen3-VL — vision-language instruct model for image captioning and binary classification checks
• Qwen3-VL-Embedding-8B — vision-language embedding model for image similarity scoring
• Qwen3-VL-Reranker-8B — vision-language reranker model for text-image relevance scoring
• SAM2.1 — Segment Anything Model 2.1 for pixel-precise entity mask extraction
• SigLIP2-so400m — multilingual vision-language backbone for body identity embeddings in character/prop calibration

---

Batch Runners

Run entire phases unattended. Each N-run-*.sh has a -detached.sh variant that runs the same script inside a tmux session and tees the log to <script>.log; the five run scripts use set -euo pipefail, stop on the first failure, and assert the marker contract after each step (.complete must exist when run.sh exits 0). 0-setup-all.sh is a plain loop without -e.

Script	Phase	Steps (the literal list each script runs)
0-setup-all.sh	Setup	Every NNN-* directory: wipes venv/, output/, markers and vendored repos, then runs download.sh, setup.sh, download_models.sh
1-run-llm.sh	Story chain (LLM + speech)	020 030 040 050 060 065 066 067 068 069 070 071 080 082 084 — not 081 (run bash run.sh in 081-character-relationships/ after 080; its output is read by 083, 100, 115, 121, 174) and not 083 (ComfyUI phase)
2-run-comfyui.sh	Image generation	083 090 092 093 094 098 (093 runs locally; with COMFYUI_BACKEND=runcomfy the ComfyUI steps run through scripts/runcomfy/run-chain.sh)
3-run-curation.sh	Dataset curation	100 102 104 (VERDA_KEEP_ALIVE=1 for 100/102)
4-run-training.sh	LoRA training	110 112 114
5-run-production.sh	Production	Every NNN-* directory with NNN ≥ 115, derived from the directory tree at start-up (today 115 → 480); see below

5-run-production.sh (pipeline-repair Phase 0, ENF-05)

./5-run-production.sh                          # full run (release): every step from 115
./5-run-production.sh 360-rough-cut 430-final-export   # dev range FIRST [LAST]; the predecessor of FIRST must carry .complete
./5-run-production-detached.sh [FIRST [LAST]]  # same, inside tmux session run-5-production; log 5-run-production.log

• Never skips and never resumes: a step that already has .complete is re-run (partial re-runs are a development activity, rule 07).
• Marker contract (every step run.sh): .running holds the PID while run.sh executes and is removed by a trap on any exit; .complete is removed at the start of a run and written only after the last script exited 0, so it describes the most recent run. Neither marker plus an ERROR block in the run.log tail means failed. Every runner stops at the first step that exits 0 without .complete, and 5-run-production.sh refuses a dev range whose FIRST predecessor lacks .complete.

---

Defect tracking

Defects are fixed where they live and verified by re-running the pipeline from the changed step forward (rule 07). The historical fix programme is docs/pipeline-repair/00-master-plan.md (phases 0–10) with its issue inventory in docs/pipeline-repair/01-issue-inventory.md; the diagnosis of a defect found on a run goes in docs/ beside it (e.g. docs/120-lora-selection-issues.md).

Root tooling

Path	Purpose
scripts/check_env_example_grammar.py	Verifies every threshold in .env.example carries a sourced trailing comment (quality-plan 03 §1); comment grammar OK on success
MODELS.json	Model inventory; step 480 generates its "Models Used" list from it
scripts/runcomfy/, scripts/runpod/, scripts/verda/	Remote backend tooling

Rules

One-line index of .kilo/rules/ (binding for every step; each file is the full text):

Rule	Title
01	Read before write — read the real CLI/argparse, wrappers, output paths and peer scripts before invoking external code
02	Use venv paths, not bare python/pip
03	No shared code between steps — duplicate, never import across step directories
04	No silent fallbacks — fail fast on a missing model/config/dependency
05	No biasing content in prompts or Field descriptions — pipeline prompt text is story-agnostic
06	Diagnose before fixing
07	Hard-fail on missing input; no backward reads; no self re-runs; own output/ only; pipeline change ⇒ full re-run
08	Do not install packages with pip or apt
09	Comments stay relevant — no change history in code comments
10	Style-boundary invariant — steps below 160 work without the film's style; includes the pixel-domain section
11	No film-global broadcast — a film-global mood field is never concatenated verbatim into a per-shot prompt
12	Unverifiable is failed — a gate that cannot run has failed, not passed

---

Documentation

• Full Overview & Pipeline Index — Master index linking every step directory, with ASCII pipeline diagram
• Quality Plan — the current programme (2026-09-03): speech rendered before the storyboard, no video-derived shot cap, every gate sourced, perceptual review, reshoots that converge; supersedes the deferred items of the repair programme
• Pipeline Repair Programme — Phases 0–10 derived from the 2026-09-01 forensic reports (landed 2026-09-02); issue inventory is the source of the registry IDs
• Tips & Best Practices — Cross-cutting advice for the entire process
• Requirements — System and model requirements
• Remote API Guide — Running with remote GPU backends
• ComfyUI Remote — Remote ComfyUI setup and configuration
• AI Toolkit Guide — LoRA training details and configuration
• LoRA Pipeline Improvement Plan — Per-phase plan for the style-boundary invariant, scoring rigor, identity, and validation-gallery work across steps 080–124 (Phase G folded the former 127/128 into 120/124)
• Archived evaluations — Story-specific evaluation documents from earlier stories, kept as evidence with an ARCHIVED header
• Qwen3-TTS Guide — Voice design and TTS pipeline
• Qwen-VL Guide — VLM scoring, reranking, and embedding
• DreamSim API — Perceptual similarity scoring setup
• Curation Details — Dataset curation methodology and scoring

---

Project Structure

my-movie-project/
├── 000-project-setup/          Step directories, one per NNN-* (each has README.md, setup.sh, run.sh)
├── 010-source-material/        ↓
├── ...                         ↓
├── 480-final-delivery/         ↓
├── .kilo/rules/                Rules 01–12 (see the Rules index above)
├── docs/                       Overview, guides, reference docs, pipeline-repair/ plan, archive/
├── img/                        Project logo/branding
├── notes/                      Generation logs, revision notes, lessons learned
├── scripts/                    env grammar checker; backend tooling (runcomfy, runpod, verda)
├── .env.example                Global configuration template (canonical variable reference)
├── .gitattributes              Git LFS tracking rules
├── .gitignore                  Ignore patterns for AI production
├── 0-setup-all.sh              Setup all step directories
├── 1-run-llm.sh                Batch: story chain (020 030 040 050 060 065 066 067 068 069 070 071 080 082 084)
├── 2-run-comfyui.sh            Batch: ComfyUI image generation (083 090 092 093 094 098)
├── 3-run-curation.sh           Batch: dataset curation (100 102 104)
├── 4-run-training.sh           Batch: LoRA training (110 112 114)
├── 5-run-production.sh         Production runner: every step >= 115, or a FIRST [LAST] dev range
├── cleanup.sh                  Remove all outputs, logs, completion markers
├── MODELS.json                 Model inventory (480 generates its "Models Used" list from it)
├── LICENSE-apache.txt          Apache 2.0 license
└── README.md                   This file

---

Per-Step Workflow

Each step from 010 onward follows this pattern:

cd NNN-step-name/
# direnv auto-activates the correct Python venv when you cd in

# First-time setup:
bash setup.sh          # Create venv, install dependencies
bash download.sh       # Download models/workflows (if present)
bash download_models.sh # Download ML models (if present)

# Run the step:
bash run.sh

# Review outputs in output/
# Commit when satisfied:
git add -A && git commit -m "NNN: Description"

Each step directory has its own isolated environment:

• venv/ — Python virtual environment (created by setup.sh)
• .python-version — pyenv Python version
• .envrc — direnv config (auto-activates venv on cd)
• pyproject.toml — Python dependencies
• setup.sh — creates venv and installs dependencies
• run.sh — removes .complete, writes .running, executes all scripts in sequence, and writes .complete only when the last script exited 0
• scripts/ — Python scripts implementing the step

Every step reads only lower-numbered steps' output/ directories and writes only its own output/ (rule 07). A pipeline change means a full re-run from the changed step forward, never a patched partial run.

---

License

Apache 2.0. See LICENSE-apache.txt.

Copyright © 2026 Jeff Moe.

Loveland, Colorado, USA