Video/Audio Media Pipeline — multi-stage Celery chain, not blocking request

Media uploads (video, audio, image) are processed through an asynchronous Celery chain, not synchronously during the upload request. Each stage is optional — the pipeline introspects the media type and configuration to decide which stages to run.

Considered Options

• Synchronous processing: Process everything (transcode, transcribe, summarize) during the upload request. Simple to implement but blocks the HTTP request for potentially hours on long videos.
• Single monolithic task: One Celery task that runs all stages sequentially. Easier to reason about but a failure at any stage requires re-running everything.
• Multi-stage Celery chain (chosen): Individual tasks (transcode_media_task, extract_audio_task, transcribe_media_task, diarize_transcript_task, summarize_media_task) composed into a dynamic chain by enqueue_media_processing(). Each stage produces output consumed by the next. A failure at any stage sets status=failed and records failed_stage and error_message on the Media model.

Why the chain was chosen

• Each stage can be independently retried without re-doing completed work.
• The chain is dynamically built per media type: images run only the thumbnail task (below), audio skips transcoding, video runs the full pipeline.
• Dedicated queues (stems, dubbing, diarization, media-import) route ML-heavy or long-running work to isolated workers that are started only if the feature is needed.

Image thumbnails (WebP variants)

Images are not "ready immediately": they run a single task (generate_image_thumbnail_task) that re-encodes the original into two WebP variants via apps.media.services.optimize — thumbnail (1200px, for banners/cover/feed) and thumbnail_small (256px, for avatars/icons/logos). Profile/space image fields set through the headless API point at the best-available variant with fallback to the original until the task completes (see apps/headless/views.py::_pick_media_file).

Consequences

• Media status is eventually consistent — the upload endpoint returns immediately, and the client must poll or use WebSocket to learn when processing is done.
• Isolated workers for stems/dubbing/diarization need their own uv run --isolated --extra ... environments because they depend on torch/ML libraries that should never be imported by the main web process.
• The Media model has many nullable status fields and error trackers — one per pipeline stage — to support granular observability.