evOo-local/docs/media.md

# Media Studio

Media Studio indexes a photo and video library, groups it into sessions and
themes, cuts videos per platform, anonymizes people and identifying marks,
applies a brand kit, and exports platform-ready bundles with tracking links and
results. It is optional: without a configured media worker it stays hidden
(`media.capabilities` reports it unconfigured) and its jobs are skipped.

Everything runs locally: ffmpeg plus local models in a sidecar. No hosted
vendor is required. A language model is used only to plan (edit plans, copy,
theme labels, teaser verdicts) and only with AI consent; rendering,
anonymization, the compliance gate and consent checks are deterministic code.

## Architecture

```
Sources ──► MediaSourceProvider ──► ingest jobs ──► analysis (sidecar) ──► index (Postgres)
 (folder, Apple Photos via desktop,                                          │
  iOS/Android upload, emulator)                                              ▼
 Agent (media_* tools) ──► sessions, themes, teaser scores, edit plan (EDL, validated)
                                                                             │
                                                                             ▼
 anonymization ──► render (ffmpeg + motion overlays) ──► gate (output re-checked) ──► bundle / publisher
                                                                             │
                                                                 tracking links ──► metrics ──► teaser learning
```

| Layer | Where | Role |
|---|---|---|
| Domain logic | `packages/media` (`@rakazo/media`) | Pure, I/O-free: EDL schema and validation, EDL → render plan and ffmpeg argv, session clustering, teaser scoring and learning, gate rules, profile loading and merge, UTM builder, metadata checks. |
| Contracts | `packages/contracts/src/media.ts` | zod DTOs, RPC shapes (`mediaRpcShapes`), agent tool inputs (`MediaToolInputSchemas`), `PlatformProfileSchema`. |
| Interfaces | `packages/adapter-kit` | `MediaSourceProvider`, `MediaInferenceProvider` (capability-based), `MediaRenderer`, `MotionGraphicsRenderer`, `MediaPublisher`, `LinkShortener`. |
| Adapters | `packages/adapters/src/media-*.ts` | Sidecar client, deterministic emulator, folder source, ffmpeg renderer, resvg motion renderer, bundle publisher, ingest/render/publish jobs, privacy, gate, marketing. `media-services.ts` is the composition root. |
| Sidecar | `infra/media-worker` | Python/FastAPI service with the models. Bound to loopback or the internal compose network, bearer token. |
| Data | `packages/db` | `Media*` Prisma models, all scoped to a space and deleted with it. |
| API | `apps/api/src/media.ts`, `media-http.ts` | RPC procedures and byte routes (thumbnails, proxies, ranged render files, bundles, chunked upload). |
| Clients | `apps/web` (also hosted by Electron), `apps/mobile`, `apps/desktop/native/photos-helper` | Library, projects, review, brand kit, results; mobile index and resumable upload; macOS Photos helper. |

Clients have no realtime subscription for media: they poll the relevant RPC
while work is in progress (for example every 2 s while a render is `queued`,
`rendering` or `gating`) and stop when it is idle.

## Setup

### Sidecar (compose profile `media`)

1. Set the generic connection in `.env` (placeholders, see `.env.example`):

   ```sh
   MEDIA_WORKER_URL=http://media-worker:8790   # bare-metal dev: http://127.0.0.1:8790
   MEDIA_WORKER_TOKEN=REPLACE_WITH_RANDOM_TOKEN  # at least 32 random characters
   ```

2. Start the stack with the profile:

   ```sh
   docker compose -f infra/compose/docker-compose.yml --profile media up -d
   ```

   The sidecar publishes no port; the API and worker reach it on the compose
   network. It mounts `./data/media` at `/data/media`, the same path the API and
   worker use as `DATA_DIR/media`, so file paths in requests resolve on both
   sides.

3. Choose the accelerator with `MEDIA_ACCELERATOR=cpu|cuda|rocm` (build arg,
   default `cpu`). The CUDA image uses the torch cu126 wheels and
   `onnxruntime-gpu`. The ROCm image runs SAM 2 on the GPU; ONNX Runtime and
   CTranslate2 have no ROCm wheels on PyPI, so SigLIP, LaMa and Whisper run on
   the CPU there. A GPU image falls back to CPU when no GPU is visible.

4. Model weights are not in the repository or the image. They download on first
   use into the `media-models` volume, are checked against pinned SHA256 hashes
   and sizes (`infra/media-worker/app/models.py`), and are deleted on mismatch.
   Warm the volume ahead of time with
   `docker compose run --rm media-worker python -m app prefetch`; set
   `MEDIA_MODEL_DOWNLOAD=0` to forbid downloads (missing weights then report the
   operation as unsupported).

Sidecar variables (`MEDIA_ROOT`, concurrency, size and time limits,
`MEDIA_OPTIONAL_BACKENDS`) are documented in `infra/media-worker/README.md`.
There is one connection for the API, never a variable per model; model choice
is sidecar configuration.

### API and worker variables

| Variable | Default | Purpose |
|---|---|---|
| `MEDIA_WORKER_URL`, `MEDIA_WORKER_TOKEN` | unset | The sidecar connection. Both set → Media Studio is enabled. |
| `MEDIA_PROFILES_DIR` | `~/.evoo/media-profiles` | Private platform profiles (below). |
| `MEDIA_GPU_CONCURRENCY` | `1` | Model-bound media jobs per worker process. |
| `MEDIA_CPU_CONCURRENCY` | `2` | ffmpeg-bound media jobs per worker process. |
| `DATA_DIR` | `./data` | Derived files live under `DATA_DIR/media`. |

The API and worker run ffmpeg and ffprobe themselves (proxies, renders,
probes); they must be on `PATH`. Hardware encoders (VAAPI, VideoToolbox, NVENC)
are used when detected, with software x264 as the fallback.

With `AGENT_RUNTIME=scripted` the deterministic emulator replaces the sidecar
and an emulator source provides synthetic clips. Tests and web E2E run this way,
offline.

### Private profiles directory

Shipped profiles live in `packages/media/profiles/`. Profiles for paid or adult
platforms, real brand assets and any marketing strategy never go into this
repository. Put them in `MEDIA_PROFILES_DIR`, outside the repo:

- Every `*.json` file there is validated against `PlatformProfileSchema`.
- A private profile with the same `id` as a shipped one replaces it.
- An invalid file is skipped whole and logged; it is never half-loaded.

In compose the API and worker run in containers, so `~` is the container's
home. Point the variable at a mounted path instead, for example
`MEDIA_PROFILES_DIR=/data/media-profiles` (that is `./data/media-profiles` on
the host, which is git-ignored).

## How the pipeline works

1. **Sources.** A folder source (only the deployment owner may add one, the same
   rule as host folders), Apple Photos through the macOS desktop helper
   (PhotoKit, macOS only; the source is not offered elsewhere), the mobile
   library index with a resumable chunked upload, and the emulator. Originals are
   referenced by source, external id and a streamed content hash, never copied.
2. **Ingest jobs** (Graphile, idempotent, resumable, cancellable):
   `media.sync` (metadata only, 500 per page, cursor based), `media.proxy`
   (720p H.264 proxy, poster and sprite, rotation applied, HDR tone-mapped,
   variable frame rate normalized), `media.analyze` (scenes, transcript with
   word times, faces and tracks, frame embeddings, content scores, loudness and
   beats, saliency), `media.cluster` (sessions, themes, face clusters). A new
   model version adds a new analysis row and never overwrites. A full disk
   pauses a job with a clear message.
3. **Sessions and themes.** A new session starts after a gap over 45 minutes
   (configurable), a jump over 2 km, or a new calendar day with a gap over 3 h;
   neighbouring sessions with mean-embedding cosine ≥ 0.85 and a gap under 2 h
   merge. Themes are agglomerative clusters of embeddings. Theme labels come
   from a model only with AI consent; otherwise they are rule based (place,
   time, album) and can be renamed.
4. **Edit plan.** The agent (or the user) writes an EDL (`edlVersion: 1`). It is
   validated against asset durations, the profile's hard duration limits, track
   overlaps, references and safe zones; errors are machine-readable so the agent
   can correct them.
5. **Render.** `compileRenderPlan` turns the EDL into argv arrays (never a shell
   string; IDs restricted to `[a-z0-9_-]`, text escaped). Sources are redacted
   first, motion overlays (intros, outros, lower thirds, captions, end cards)
   are pre-rendered from brand-kit-only templates, then ffmpeg renders a draft
   from proxies or the final from originals, with two-pass `loudnorm` to the
   profile target (default −14 LUFS, −1 dBTP).
6. **Gate.** Before any export, on the rendered output: technical probe,
   content classifier at the profile's sample rate plus every cut, faces
   re-detected and matched to people, pixel-level check that every protected
   region is covered, consent and age proof where required, text against
   banned patterns, link policy and lengths, and AI-alteration disclosure. Any
   failure is `blocked`. The agent may repair a blocked render only with
   compliance-preserving changes (trim or drop a clip, other segment, other
   crop, wider cover) for at most 3 rounds; then the user takes over.
7. **Marketing.** Three thumbnail variants (also gated), per-profile copy
   checked against the profile, UTM tracking links (no shortener by default),
   and an export bundle: video, posters, SRT/VTT, `caption.txt`,
   `metadata.json`, `UPLOAD-CHECKLIST.md`. Paid platforms only ever get the
   bundle.
8. **Scheduling and results.** A scheduled publication is a `media.publish` job
   with a run time; at that time the bundle is built, handed to the publisher
   (by default the bundle itself) and the author gets a push to upload it.
   Uploads to connected accounts go through the agent's connector tools, where
   the existing approval rules apply. Metrics come from CSV import
   with saved column mappings or manual entry and feed teaser learning, which
   starts after 20 publications.

## Platform profiles

A profile (`PlatformProfileSchema` in `packages/contracts/src/media.ts`) is one
JSON file:

| Field | Meaning |
|---|---|
| `id`, `name`, `kind` | `id` matches `[a-z0-9_-]`; `kind` is `free_social`, `paid_video`, `own_site` or `messenger`. |
| `sourceUrls`, `reviewedAt`, `unverified` | Where the values come from, when they were reviewed, and whether any value is unconfirmed. The UI flags unverified profiles and those reviewed more than 90 days ago. |
| `video` | Aspect ratios, width, height, fps, codec (`h264`, `hevc`), max bitrate, max file size. |
| `duration` | `hardMinMs`, `hardMaxMs`, `recommended[]`, `hookWindowMs`. |
| `safeZones` | Margins in percent kept free of overlays. |
| `captions` | Recommended, required, max lines. |
| `audio` | Target LUFS, true peak, music policy. |
| `text` | Title and caption length, hashtag limit, `linkPolicy`, `bannedPatterns[]` (case-insensitive regex with a reason). |
| `content` | Max classifier score per category, sample fps, `blockOn`. |
| `disclosure` | AI-altered label, paid-partnership label, age-restricted flag. |
| `consent` | Performer consent and age verification. A `paid_video` profile always requires both, whatever the file says. |
| `export` | Bundle contents and an optional preview clip length. |

Shipped profiles:

| File | Notes |
|---|---|
| `youtube-shorts`, `youtube`, `tiktok`, `instagram-reels`, `instagram-feed-4x5`, `facebook-reels`, `x-twitter`, `linkedin`, `snapchat-spotlight`, `pinterest-video-pin` | Conservative defaults for public social platforms. All are `unverified: true`: the official pages in `sourceUrls` could not be fetched while these files were written, so values come from previously published documentation and are set on the strict side. Review each against its sources and clear the flag, or override it privately. Pinterest's former Idea Pins are now regular video Pins. |
| `example-short-vertical`, `example-landscape` | Neutral examples. |
| `example-paid-video` | Neutral placeholder for a paid video platform: consent and age proof required, age-restricted flag set. |
| `own-site-hls` | Your own site. The render is a progressive MP4; HLS packaging is left to the site. |

## Privacy

- **Biometric data stays local.** Face detections, tracks, face embeddings and
  person clusters are computed by the local sidecar or emulator and stored in
  the local database and `DATA_DIR/media`. They are never sent to an external
  service, with or without AI consent.
- **AI consent.** Any media sent to a non-local model (theme labels from cover
  thumbnails, teaser verdicts, copy) passes the existing AI-consent gate.
  Without consent the deterministic path is used.
- **Consent and age proof** for performers can only be recorded by the signed-in
  user in the UI. Agent tools cannot write them.
- **Deletion.** Deleting a person removes their face tracks, embeddings, marks
  and mask files. Deleting a space cascades to every media row; derived files
  live under `DATA_DIR/media/<spaceId>/`.
- **Metadata stripping.** Every export drops container and stream metadata
  (`-map_metadata -1`, bitexact muxing), so GPS, device serials and encoder
  software are removed and only allowed fields (title) are set again. An
  ffprobe check after the render proves it; otherwise the render fails.
  Posters are extracted without metadata and thumbnail variants are written as
  PNG without EXIF or text chunks.
- **Locations** are stored rounded.
- **Paths.** Every file path goes through one function that rejects traversal,
  absolute inputs and symlinks leaving the media root. The sidecar enforces the
  same rule for `MEDIA_ROOT`.

What it does not guarantee: anonymization is only as good as detection. Low
confidence keeps the last mask and enlarges it, detection gaps over 500 ms
block the export, and the gate re-checks the output, but a person the detector
never sees (back of the head, heavy occlusion) is not recognised as a person.
Voice alteration (pitch or formant) obscures a voice; it is not anonymization.
Review the output before publishing.

## Hard limits

These are not built and must not be added:

- Anything that evades platform moderation or classifiers (adversarial
  overlays, inserted frames, colour or noise tricks, age-gate bypass).
  "Compliance" means the content actually follows the rules.
- Face swaps with a real third person. Only a synthetic, non-existent identity
  is allowed, and the API accepts no target-face image.
- Export to paid platforms without documented consent and age proof of every
  recognised participant. Unmatched faces block.
- Uploading biometric data to external services.

Failures fail closed: if anonymization, the gate or the consent check cannot
show a safe state, nothing is exported.

## Licenses

Checked October 2026 against the upstream license files listed in
`infra/media-worker/LICENSES.md` and the package metadata installed in this
repository. "Commercial use" means the code or weights license permits it;
training-data terms were not reviewed legally. No non-commercial model is a
default. Weights are downloaded at runtime and never shipped.

### Models

| Name | Purpose | License | Commercial use | Default / optional |
|---|---|---|---|---|
| OpenCV YuNet 2023mar | Face detection, 5-point landmarks, tracking input | MIT | yes | default |
| Whisper small (faster-whisper / CTranslate2 conversion) | Transcription with word timestamps | MIT | yes | default |
| SigLIP base patch16-224 (ONNX) | Frame embeddings | Apache-2.0 | yes | default |
| SigLIP base patch16-224 (ONNX), zero-shot prompts | Content classifier | Apache-2.0 | yes | default, experimental |
| SAM 2.1 Hiera tiny | Prompted segmentation with video propagation | Apache-2.0 | yes | default, experimental |
| Big-LaMa (ONNX) + optical flow | Per-frame inpainting with temporal smoothing | Apache-2.0 | yes | default, experimental |
| OpenCV SFace 2021dec | Face identity embeddings for local person clustering | Apache-2.0 weights; training-data provenance unverified | unclear | optional (`MEDIA_OPTIONAL_BACKENDS=faceIdentity`) |
| Synthetic identity generator and swap | `synthesizeIdentity` | none found with a commercial license | — | unavailable; always reports unsupported |
| ProPainter | Video inpainting | S-Lab License 1.0 (non-commercial) | no | not used |
| InsightFace model zoo (SCRFD, ArcFace, inswapper) | Detection, recognition, swap | code MIT, models non-commercial | no | not used |
| NudeNet v3 | Nudity detection | AGPL-3.0 | copyleft | not used |
| MediaPipe Face Landmarker | Dense landmarks | Apache-2.0 | yes | not used |
| OpenCLIP | Embeddings | MIT | yes | not used |

### Methods and tools

| Name | Purpose | License | Commercial use | Default / optional |
|---|---|---|---|---|
| FFmpeg / ffprobe | Decode, render, loudness, probes | LGPL-2.1+ or GPL-2+ depending on the build | yes, run as a separate process | default |
| PySceneDetect (AdaptiveDetector) | Shot detection | BSD-3-Clause | yes | default |
| librosa | Beats and tempo | ISC | yes | default |
| IoU tracker, spectral-residual saliency, redaction | Tracking, saliency, blur/pixelate/cover | this repository (Apache-2.0) | yes | default |
| `@resvg/resvg-js` 2.6.2 | Motion overlays: SVG frames rasterized, then encoded by ffmpeg | MPL-2.0 | yes, used unmodified as a library | default |
| `sharp` 0.35.4 (bundles libvips) | Thumbnail and poster processing | Apache-2.0; prebuilt libvips LGPL-3.0-or-later | yes, dynamically linked | default |
| `expo-media-library`, `expo-video` | Mobile library index and preview | MIT | yes | default on mobile |
| PhotoKit | Apple Photos access in the desktop helper | Apple system framework | yes, on macOS | default on macOS |

### Sidecar libraries

FastAPI (MIT), Uvicorn (BSD-3-Clause), Pydantic (MIT), NumPy (BSD-3-Clause),
OpenCV `opencv-python-headless` (Apache-2.0, wheel bundles LGPL FFmpeg),
soundfile (BSD-3-Clause), PyTorch and torchvision (BSD-3-Clause), ONNX Runtime
(MIT), faster-whisper (MIT), CTranslate2 (MIT), Hugging Face tokenizers
(Apache-2.0), `sam2` (Apache-2.0), Hydra and iopath (MIT). All are commercially
usable.

### Remotion

The spec proposed Remotion for motion graphics. It is not used: Remotion
requires a paid company license above a team-size threshold and needs headless
Chromium. The motion renderer instead draws each template as an SVG per frame,
rasterizes it with resvg and encodes a QuickTime Animation (alpha) clip with
ffmpeg, behind the same `MotionGraphicsRenderer` interface.

The license entries above were taken from the upstream license files as
recorded in `infra/media-worker/LICENSES.md`, the pinned registry in
`infra/media-worker/app/models.py` and the installed npm package metadata. The
upstream pages themselves could not be fetched again while this document was
written; re-check them before relying on this table for a release.

## Known limitations

- **Content classifier is experimental.** It is zero-shot SigLIP with text
  prompts, not a trained moderation model. Treat its scores as a first filter;
  it does not replace a platform's own review. Segmentation and inpainting are
  experimental too, and inpainting with a low quality score falls back to a
  cover or reframe suggestion.
- **Synthetic identity is unavailable.** No commercially licensed model was
  found, so the mode reports "not available".
- **CUDA and ROCm images are unbuilt in CI.** Only the CPU image is built and
  tested; the GPU variants are defined but untested.
- **Job limits are per worker process.** `MEDIA_GPU_CONCURRENCY` and
  `MEDIA_CPU_CONCURRENCY` bound one worker; several workers multiply them. The
  sidecar's own gates (`MEDIA_GPU_CONCURRENCY`, `MEDIA_QUEUE_SIZE`) still cap
  inference and answer `busy` when full.
- **Motion graphics use resvg, not Remotion** (see above): templates are SVG,
  so effects needing a browser engine (CSS, WebGL) are not available.
- **Scheduled publishing** is a `media.publish` background job with a run time,
  not a Routine. Rescheduling replaces the job and cancelling removes it.
- **Platform profiles are unverified** until someone checks them against the
  official pages listed in each file.
- **Own site output** is a progressive MP4; no HLS ladder is produced.
- **Apple Photos** needs the macOS desktop app; the source is not offered on
  Windows, Linux or the web.