Chinese Art

Public-domain and freely-licensed Chinese graphic art (paintings, drawings, calligraphy, maps, woodblock prints, cave murals, etc.) sourced from Wikimedia Commons and curated Wikipedia article galleries, organized by historical period for printing.

Pipeline

Two stages, both runnable concurrently:

1. scrape.py — walks seed categories/articles, dedups by sha1, applies a mime/mediatype gate, downloads accepted files into pending/, and inserts one row per accepted file into state/state.db (rows table). Structural rejects go into the structural_rejects table. Per-seed continue-cursors live in the cursors table.

2. enrich.py — drains pending rows from state.db via a streaming feeder/worker pool sized to the total VLM concurrency (len(API_URLS) * VLM_CONCURRENCY). Each row is rasterized, sent to an OpenAI-compatible VLM endpoint, and updated in place in state.db with the VLM's verdict (period, english_title, english_description, …). Accepted binaries move to dynasties/<period>/; VLM-rejected binaries move to dynasties/<period>/rejected/. Use --watch to keep draining as scrape.py adds new rows.

State

All persistent state is in two SQLite databases (WAL mode; one writer plus many readers without external locking):

• state/state.db — pipeline state: rows (one row per file across pending + enriched + VLM-rejected), structural_rejects (Stage 1 tombstones), cursors (per-seed continue state).
• state/wp_cache.db — read-only Wikipedia article-extract cache populated by scrape.py and consumed by enrich.py.