AI Movie Production Process

A complete, step-by-step pipeline for transforming short stories into fully produced AI-generated films using open source tools.

---

Quick Start

# Clone this template to start a new movie project
git clone https://spacecruft.org/movies/process my-movie-project
cd my-movie-project

# Initialize Git LFS (one-time per machine)
git lfs install

# Copy and configure environment
cp .env.example .env
nano .env   # Set API URLs, model paths, GPU config

# Start at step 000
cat 000-project-setup/README.md

Then work through each numbered directory in order. Each directory has a README.md explaining exactly what to do, what inputs you need, and what outputs to produce.

---

What's Inside

69 step directories organized into 13 phases with 6 GPU swap points:

Phase	Steps	Description
Pre-Production — Story & Script	000–080	Source story, extract characters/locations, write screenplay, create storyboard, define style guide
Pre-Production — Visual Prompts	082–086	LLM extracts visual descriptions for characters, locations, props
🔄 GPU Swap 1: Unload LLM → Load ComfyUI
Pre-Production — Visual Design	090–098	Generate design images via ComfyUI for characters, locations (PyraCanny ControlNet), props
🔄 GPU Swap 2: Unload ComfyUI → Load Qwen3-VL
Pre-Production — Dataset Curation	100–104	VLM rate (Qwen3-VL-Reranker + DreamSim/Embedding), curate, and caption training images
🔄 GPU Swap 3: Unload Qwen3-VL → Load AI Toolkit
Pre-Production — LoRA Training	110–114	Train Qwen-Image LoRAs for characters, locations, props (AI Toolkit)
🔄 GPU Swap 4: Unload AI Toolkit → Load ComfyUI
Pre-Production — LoRA Validation	120–124	Automated checkpoint + strength selection via DreamSim + Reranker scoring, visual validation galleries
🔄 GPU Swap 5: Unload ComfyUI → Load LLM
Production — Shot Planning	160	LLM maps shots to LoRAs, compile per-scene mood boards
🔄 GPU Swap 6: Unload LLM → Load ComfyUI
Production — Keyframe Generation	170–180	Scout generation, GDINO/SAM2 segmentation, layered inpainting (location → character → prop), harmonization, face refinement, keyframe selection, outpainting
Production — Voice & Dialogue	189–210	Qwen3-TTS voice design, dialogue generation, narration (audio before video — LongCat requires dialogue as input)
Production — Video Generation	220–270	LongCat-Video-B200 video clips with lip sync, VLM selection, upscale, finalize
Production — Audio: Sound & Music	300–350	FreeSound CC0 retrieval, MOSS-SoundEffect SFX generation, Qwen3-Omni SFX selection, ACE-Step 1.5 XL Turbo music composition
Post-Production — Assembly & Finishing	360–430	Edit, mix, grade, titles, credits, subtitles, export
Review & Delivery	440–480	Quality review, revisions, feedback, ship

Key Tools & Infrastructure

• LLM — OpenAI-compatible API for all text analysis, screenplay, storyboard, prompt extraction, and shot planning
• ComfyUI — Image generation with multi-GPU parallel dispatch (COMFYUI_API_URLS), PyraCanny ControlNet for location consistency
• AI Toolkit — LoRA training for Qwen-Image models (characters, locations, props) with parallel GPU training
• Qwen3-VL — Dataset curation and LoRA validation (Reranker-8B for text relevance, Embedding-8B for cluster consistency, 30B-A3B-Instruct for captioning)
• DreamSim — Perceptual similarity scoring (CLIP + OpenCLIP + DINO ViT-B/16 ensemble) for curation, hero selection, and LoRA validation
• Grounding DINO + SAM2.1 — Text-guided object detection and pixel-precise segmentation for keyframe entity masking
• LongCat-Video-B200 — Video generation with audio-driven lip sync and multi-character support
• Qwen3-TTS — Voice design (VoiceDesign model) and dialogue generation (Base model, ICL cloning)
• FreeSound — CC0 sound effects retrieval via API with LLM-generated search queries
• MOSS-SoundEffect — Text-to-audio SFX generation (8B, duration-controlled)
• Qwen3-Omni — Audio understanding model (30B-A3B MoE) for SFX candidate evaluation and selection
• ACE-Step 1.5 XL Turbo — AI music composition (4B DiT + 5Hz LM, theme, scene scores, stingers)
• Configuration — All settings via root .env.example (API URLs, model paths, GPU IDs, training hyperparameters)

Models

• ACE-Step 1.5 XL Turbo DiT — AI music composition diffusion transformer
• ACE-Step 5Hz LM — AI music composition language model
• AuraFace-v1 — face identity embedding model used for character identity calibration
• CLIP ViT-B/16 — vision-language backbone used in DreamSim ensemble
• DINO ViT-B/16 — self-supervised vision backbone used in DreamSim ensemble
• DINOv2-giant — self-supervised vision backbone for body identity embeddings in character/prop calibration
• DreamSim — perceptual similarity ensemble for image-pair distance scoring
• GFPGAN v1.4 — face restoration for keyframe refinement
• GLM 5.1 — text LLM for story analysis, screenplay, storyboard, prompt extraction, shot planning
• Grounding DINO — text-guided object detection for entity segmentation
• LibreHPS-4B v1.1 — aesthetic / human-preference scorer
• LongCat-Video — image-to-video generation model
• LongCat-Video-Avatar-1.5 — avatar video generation with audio-driven lip sync
• MOSS-SoundEffect — text-to-audio sound effect generation
• OpenCLIP ViT-B/16 — vision-language backbone used in DreamSim ensemble
• Qwen-2.5-VL-7B — CLIP text encoder for Qwen-Image
• Qwen-Image — image generation diffusion model
• Qwen-Image ControlNet-Union — ControlNet for location design + character/prop inpainting
• Qwen-Image-Edit — image editing diffusion model for outpainting
• Qwen-Image VAE — variational autoencoder for Qwen-Image
• Qwen3-Omni — audio understanding model for SFX evaluation and selection
• Qwen3-TTS Base — text-to-speech base model for dialogue and narration with ICL voice cloning
• Qwen3-TTS VoiceDesign — text-to-speech voice design model for voice profile creation
• Qwen3-VL — vision-language instruct model for image captioning and binary classification checks
• Qwen3-VL-Embedding-8B — vision-language embedding model for image similarity scoring
• Qwen3-VL-Reranker-8B — vision-language reranker model for text-image relevance scoring
• SAM2.1 — Segment Anything Model 2.1 for pixel-precise entity mask extraction
• SigLIP2-so400m — multilingual vision-language backbone for body identity embeddings in character/prop calibration

---

Batch Runners

Run entire phases unattended (each has a -detached.sh variant for tmux/SSH):

Script	Phase	Steps
0-setup-all.sh	Setup	Download + setup all 69 step directories
1-run-llm.sh	LLM extraction	020–086 (story analysis, screenplay, visual prompts)
2-run-comfyui.sh	Image generation	090–098 (character/location/prop design)
3-run-curation.sh	Dataset curation	100–104 (rate, curate, caption)
4-run-training.sh	LoRA training	110–114 (character/location/prop LoRAs)

---

Documentation

• Full Overview & Pipeline Index — Master index linking all 69 steps with ASCII pipeline diagram
• Tips & Best Practices — Cross-cutting advice for the entire process
• Requirements — System and model requirements
• Remote API Guide — Running with remote GPU backends
• ComfyUI Remote — Remote ComfyUI setup and configuration
• AI Toolkit Guide — LoRA training details and configuration
• LoRA Pipeline Improvement Plan — Per-phase plan for the style-boundary invariant, scoring rigor, identity, and validation-gallery work across steps 080–127
• Qwen3-TTS Guide — Voice design and TTS pipeline
• Qwen-VL Guide — VLM scoring, reranking, and embedding
• DreamSim API — Perceptual similarity scoring setup
• Curation Details — Dataset curation methodology and scoring

---

Project Structure

my-movie-project/
├── 000-project-setup/          69 step directories (each has README.md, setup.sh, run.sh)
├── 010-source-material/        ↓
├── ...                         ↓
├── 480-final-delivery/         ↓
├── docs/                       Overview, guides, and reference docs
├── img/                        Project logo/branding
├── notes/                      Generation logs, revision notes, lessons learned
├── scripts/                    Backend tooling (runcomfy, runpod, verda)
├── .env.example                Global configuration template
├── .gitattributes              Git LFS tracking rules
├── .gitignore                  Ignore patterns for AI production
├── 0-setup-all.sh              Setup all step directories
├── 1-run-llm.sh                Batch: LLM extraction phase (020–086)
├── 2-run-comfyui.sh            Batch: ComfyUI image generation (090–098)
├── 3-run-curation.sh           Batch: dataset curation (100–104)
├── 4-run-training.sh           Batch: LoRA training (110–114)
├── cleanup.sh                  Remove all outputs, logs, completion markers
├── LICENSE-apache.txt          Apache 2.0 license
└── README.md                   This file

---

Per-Step Workflow

Each step from 010 onward follows this pattern:

cd NNN-step-name/
# direnv auto-activates the correct Python venv when you cd in

# First-time setup:
bash setup.sh          # Create venv, install dependencies
bash download.sh       # Download models/workflows (if present)
bash download_models.sh # Download ML models (if present)

# Run the step:
bash run.sh

# Review outputs in output/
# Commit when satisfied:
git add -A && git commit -m "NNN: Description"

Each step directory has its own isolated environment:

• venv/ — Python virtual environment (created by setup.sh)
• .python-version — pyenv Python version
• .envrc — direnv config (auto-activates venv on cd)
• pyproject.toml — Python dependencies
• setup.sh — creates venv and installs dependencies
• run.sh — executes all scripts in sequence
• scripts/ — Python scripts implementing the step

---

License

Apache 2.0. See LICENSE-apache.txt.

Copyright © 2026 Jeff Moe.

Loveland, Colorado, USA