evOo-local/docs/media.md

Media Studio

Media Studio indexes a photo and video library, groups it into sessions and themes, cuts videos per platform, anonymizes people and identifying marks, applies a brand kit, and exports platform-ready bundles with tracking links and results. It is optional: without a configured media worker it stays hidden (media.capabilities reports it unconfigured) and its jobs are skipped.

Everything runs locally: ffmpeg plus local models in a sidecar. No hosted vendor is required. A language model is used only to plan (edit plans, copy, theme labels, teaser verdicts) and only with AI consent; rendering, anonymization, the compliance gate and consent checks are deterministic code.

Architecture

Sources ──► MediaSourceProvider ──► ingest jobs ──► analysis (sidecar) ──► index (Postgres)
 (folder, Apple Photos via desktop,                                          │
  iOS/Android upload, emulator)                                              ▼
 Agent (media_* tools) ──► sessions, themes, teaser scores, edit plan (EDL, validated)
                                                                             │
                                                                             ▼
 anonymization ──► render (ffmpeg + motion overlays) ──► gate (output re-checked) ──► bundle / publisher
                                                                             │
                                                                 tracking links ──► metrics ──► teaser learning

Layer	Where	Role
Domain logic	packages/media (@rakazo/media)	Pure, I/O-free: EDL schema and validation, EDL → render plan and ffmpeg argv, session clustering, teaser scoring and learning, gate rules, profile loading and merge, UTM builder, metadata checks.
Contracts	packages/contracts/src/media.ts	zod DTOs, RPC shapes (mediaRpcShapes), agent tool inputs (MediaToolInputSchemas), PlatformProfileSchema.
Interfaces	packages/adapter-kit	MediaSourceProvider, MediaInferenceProvider (capability-based), MediaRenderer, MotionGraphicsRenderer, MediaPublisher, LinkShortener.
Adapters	packages/adapters/src/media-*.ts	Sidecar client, deterministic emulator, folder source, ffmpeg renderer, resvg motion renderer, bundle publisher, ingest/render/publish jobs, privacy, gate, marketing. media-services.ts is the composition root.
Sidecar	infra/media-worker	Python/FastAPI service with the models. Bound to loopback or the internal compose network, bearer token.
Data	packages/db	Media* Prisma models, all scoped to a space and deleted with it.
API	apps/api/src/media.ts, media-http.ts	RPC procedures and byte routes (thumbnails, proxies, ranged render files, bundles, chunked upload).
Clients	apps/web (also hosted by Electron), apps/mobile, apps/desktop/native/photos-helper	Library, projects, review, brand kit, results; mobile index and resumable upload; macOS Photos helper.

Clients have no realtime subscription for media: they poll the relevant RPC while work is in progress (for example every 2 s while a render is queued, rendering or gating) and stop when it is idle.

Setup

Sidecar (compose profile media)

1. Set the generic connection in .env (placeholders, see .env.example):

   MEDIA_WORKER_URL=http://media-worker:8790   # bare-metal dev: http://127.0.0.1:8790
   MEDIA_WORKER_TOKEN=REPLACE_WITH_RANDOM_TOKEN  # at least 32 random characters
   

2. Start the stack with the profile:

   docker compose -f infra/compose/docker-compose.yml --profile media up -d
   

   The sidecar publishes no port; the API and worker reach it on the compose network. It mounts ./data/media at /data/media, the same path the API and worker use as DATA_DIR/media, so file paths in requests resolve on both sides.

3. Choose the accelerator with MEDIA_ACCELERATOR=cpu|cuda|rocm (build arg, default cpu). The CUDA image uses the torch cu126 wheels and onnxruntime-gpu. The ROCm image runs SAM 2 on the GPU; ONNX Runtime and CTranslate2 have no ROCm wheels on PyPI, so SigLIP, LaMa and Whisper run on the CPU there. A GPU image falls back to CPU when no GPU is visible.

4. Model weights are not in the repository or the image. They download on first use into the media-models volume, are checked against pinned SHA256 hashes and sizes (infra/media-worker/app/models.py), and are deleted on mismatch. Warm the volume ahead of time with docker compose run --rm media-worker python -m app prefetch; set MEDIA_MODEL_DOWNLOAD=0 to forbid downloads (missing weights then report the operation as unsupported).

Sidecar variables (MEDIA_ROOT, concurrency, size and time limits, MEDIA_OPTIONAL_BACKENDS) are documented in infra/media-worker/README.md. There is one connection for the API, never a variable per model; model choice is sidecar configuration.

API and worker variables

Variable	Default	Purpose
MEDIA_WORKER_URL, MEDIA_WORKER_TOKEN	unset	The sidecar connection. Both set → Media Studio is enabled.
MEDIA_PROFILES_DIR	~/.evoo/media-profiles	Private platform profiles (below).
MEDIA_GPU_CONCURRENCY	1	Model-bound media jobs per worker process.
MEDIA_CPU_CONCURRENCY	2	ffmpeg-bound media jobs per worker process.
DATA_DIR	./data	Derived files live under DATA_DIR/media.

The API and worker run ffmpeg and ffprobe themselves (proxies, renders, probes); they must be on PATH. Hardware encoders (VAAPI, VideoToolbox, NVENC) are used when detected, with software x264 as the fallback.

With AGENT_RUNTIME=scripted the deterministic emulator replaces the sidecar and an emulator source provides synthetic clips. Tests and web E2E run this way, offline.

Private profiles directory

Shipped profiles live in packages/media/profiles/. Profiles for paid or adult platforms, real brand assets and any marketing strategy never go into this repository. Put them in MEDIA_PROFILES_DIR, outside the repo:

• Every *.json file there is validated against PlatformProfileSchema.
• A private profile with the same id as a shipped one replaces it.
• An invalid file is skipped whole and logged; it is never half-loaded.

In compose the API and worker run in containers, so ~ is the container's home. Point the variable at a mounted path instead, for example MEDIA_PROFILES_DIR=/data/media-profiles (that is ./data/media-profiles on the host, which is git-ignored).

How the pipeline works

1. Sources. A folder source (only the deployment owner may add one, the same rule as host folders), Apple Photos through the macOS desktop helper (PhotoKit, macOS only; the source is not offered elsewhere), the mobile library index with a resumable chunked upload, and the emulator. Originals are referenced by source, external id and a streamed content hash, never copied.
2. Ingest jobs (Graphile, idempotent, resumable, cancellable): media.sync (metadata only, 500 per page, cursor based), media.proxy (720p H.264 proxy, poster and sprite, rotation applied, HDR tone-mapped, variable frame rate normalized), media.analyze (scenes, transcript with word times, faces and tracks, frame embeddings, content scores, loudness and beats, saliency), media.cluster (sessions, themes, face clusters). A new model version adds a new analysis row and never overwrites. A full disk pauses a job with a clear message.
3. Sessions and themes. A new session starts after a gap over 45 minutes (configurable), a jump over 2 km, or a new calendar day with a gap over 3 h; neighbouring sessions with mean-embedding cosine ≥ 0.85 and a gap under 2 h merge. Themes are agglomerative clusters of embeddings. Theme labels come from a model only with AI consent; otherwise they are rule based (place, time, album) and can be renamed.
4. Edit plan. The agent (or the user) writes an EDL (edlVersion: 1). It is validated against asset durations, the profile's hard duration limits, track overlaps, references and safe zones; errors are machine-readable so the agent can correct them.
5. Render. compileRenderPlan turns the EDL into argv arrays (never a shell string; IDs restricted to [a-z0-9_-], text escaped). Sources are redacted first, motion overlays (intros, outros, lower thirds, captions, end cards) are pre-rendered from brand-kit-only templates, then ffmpeg renders a draft from proxies or the final from originals, with two-pass loudnorm to the profile target (default −14 LUFS, −1 dBTP).
6. Gate. Before any export, on the rendered output: technical probe, content classifier at the profile's sample rate plus every cut, faces re-detected and matched to people, pixel-level check that every protected region is covered, consent and age proof where required, text against banned patterns, link policy and lengths, and AI-alteration disclosure. Any failure is blocked. The agent may repair a blocked render only with compliance-preserving changes (trim or drop a clip, other segment, other crop, wider cover) for at most 3 rounds; then the user takes over.
7. Marketing. Three thumbnail variants (also gated), per-profile copy checked against the profile, UTM tracking links (no shortener by default), and an export bundle: video, posters, SRT/VTT, caption.txt, metadata.json, UPLOAD-CHECKLIST.md. Paid platforms only ever get the bundle.
8. Scheduling and results. A scheduled publication is a media.publish job with a run time; at that time the bundle is built, handed to the publisher (by default the bundle itself) and the author gets a push to upload it. Uploads to connected accounts go through the agent's connector tools, where the existing approval rules apply. Metrics come from CSV import with saved column mappings or manual entry and feed teaser learning, which starts after 20 publications.

Platform profiles

A profile (PlatformProfileSchema in packages/contracts/src/media.ts) is one JSON file:

Field	Meaning
id, name, kind	id matches [a-z0-9_-]; kind is free_social, paid_video, own_site or messenger.
sourceUrls, reviewedAt, unverified	Where the values come from, when they were reviewed, and whether any value is unconfirmed. The UI flags unverified profiles and those reviewed more than 90 days ago.
video	Aspect ratios, width, height, fps, codec (h264, hevc), max bitrate, max file size.
duration	hardMinMs, hardMaxMs, recommended[], hookWindowMs.
safeZones	Margins in percent kept free of overlays.
captions	Recommended, required, max lines.
audio	Target LUFS, true peak, music policy.
text	Title and caption length, hashtag limit, linkPolicy, bannedPatterns[] (case-insensitive regex with a reason).
content	Max classifier score per category, sample fps, blockOn.
disclosure	AI-altered label, paid-partnership label, age-restricted flag.
consent	Performer consent and age verification. A paid_video profile always requires both, whatever the file says.
export	Bundle contents and an optional preview clip length.

Shipped profiles:

File	Notes
youtube-shorts, youtube, tiktok, instagram-reels, instagram-feed-4x5, facebook-reels, x-twitter, linkedin, snapchat-spotlight, pinterest-video-pin	Conservative defaults for public social platforms. All are unverified: true: the official pages in sourceUrls could not be fetched while these files were written, so values come from previously published documentation and are set on the strict side. Review each against its sources and clear the flag, or override it privately. Pinterest's former Idea Pins are now regular video Pins.
example-short-vertical, example-landscape	Neutral examples.
example-paid-video	Neutral placeholder for a paid video platform: consent and age proof required, age-restricted flag set.
own-site-hls	Your own site. The render is a progressive MP4; HLS packaging is left to the site.

Privacy

• Biometric data stays local. Face detections, tracks, face embeddings and person clusters are computed by the local sidecar or emulator and stored in the local database and DATA_DIR/media. They are never sent to an external service, with or without AI consent.
• AI consent. Any media sent to a non-local model (theme labels from cover thumbnails, teaser verdicts, copy) passes the existing AI-consent gate. Without consent the deterministic path is used.
• Consent and age proof for performers can only be recorded by the signed-in user in the UI. Agent tools cannot write them.
• Deletion. Deleting a person removes their face tracks, embeddings, marks and mask files. Deleting a space cascades to every media row; derived files live under DATA_DIR/media/<spaceId>/.
• Metadata stripping. Every export drops container and stream metadata (-map_metadata -1, bitexact muxing), so GPS, device serials and encoder software are removed and only allowed fields (title) are set again. An ffprobe check after the render proves it; otherwise the render fails. Posters are extracted without metadata and thumbnail variants are written as PNG without EXIF or text chunks.
• Locations are stored rounded.
• Paths. Every file path goes through one function that rejects traversal, absolute inputs and symlinks leaving the media root. The sidecar enforces the same rule for MEDIA_ROOT.

What it does not guarantee: anonymization is only as good as detection. Low confidence keeps the last mask and enlarges it, detection gaps over 500 ms block the export, and the gate re-checks the output, but a person the detector never sees (back of the head, heavy occlusion) is not recognised as a person. Voice alteration (pitch or formant) obscures a voice; it is not anonymization. Review the output before publishing.

Hard limits

These are not built and must not be added:

• Anything that evades platform moderation or classifiers (adversarial overlays, inserted frames, colour or noise tricks, age-gate bypass). "Compliance" means the content actually follows the rules.
• Face swaps with a real third person. Only a synthetic, non-existent identity is allowed, and the API accepts no target-face image.
• Export to paid platforms without documented consent and age proof of every recognised participant. Unmatched faces block.
• Uploading biometric data to external services.

Failures fail closed: if anonymization, the gate or the consent check cannot show a safe state, nothing is exported.

Licenses

Checked October 2026 against the upstream license files listed in infra/media-worker/LICENSES.md and the package metadata installed in this repository. "Commercial use" means the code or weights license permits it; training-data terms were not reviewed legally. No non-commercial model is a default. Weights are downloaded at runtime and never shipped.

Models

Name	Purpose	License	Commercial use	Default / optional
OpenCV YuNet 2023mar	Face detection, 5-point landmarks, tracking input	MIT	yes	default
Whisper small (faster-whisper / CTranslate2 conversion)	Transcription with word timestamps	MIT	yes	default
SigLIP base patch16-224 (ONNX)	Frame embeddings	Apache-2.0	yes	default
SigLIP base patch16-224 (ONNX), zero-shot prompts	Content classifier	Apache-2.0	yes	default, experimental
SAM 2.1 Hiera tiny	Prompted segmentation with video propagation	Apache-2.0	yes	default, experimental
Big-LaMa (ONNX) + optical flow	Per-frame inpainting with temporal smoothing	Apache-2.0	yes	default, experimental
OpenCV SFace 2021dec	Face identity embeddings for local person clustering	Apache-2.0 weights; training-data provenance unverified	unclear	optional (MEDIA_OPTIONAL_BACKENDS=faceIdentity)
Synthetic identity generator and swap	synthesizeIdentity	none found with a commercial license	—	unavailable; always reports unsupported
ProPainter	Video inpainting	S-Lab License 1.0 (non-commercial)	no	not used
InsightFace model zoo (SCRFD, ArcFace, inswapper)	Detection, recognition, swap	code MIT, models non-commercial	no	not used
NudeNet v3	Nudity detection	AGPL-3.0	copyleft	not used
MediaPipe Face Landmarker	Dense landmarks	Apache-2.0	yes	not used
OpenCLIP	Embeddings	MIT	yes	not used

Methods and tools

Name	Purpose	License	Commercial use	Default / optional
FFmpeg / ffprobe	Decode, render, loudness, probes	LGPL-2.1+ or GPL-2+ depending on the build	yes, run as a separate process	default
PySceneDetect (AdaptiveDetector)	Shot detection	BSD-3-Clause	yes	default
librosa	Beats and tempo	ISC	yes	default
IoU tracker, spectral-residual saliency, redaction	Tracking, saliency, blur/pixelate/cover	this repository (Apache-2.0)	yes	default
@resvg/resvg-js 2.6.2	Motion overlays: SVG frames rasterized, then encoded by ffmpeg	MPL-2.0	yes, used unmodified as a library	default
sharp 0.35.4 (bundles libvips)	Thumbnail and poster processing	Apache-2.0; prebuilt libvips LGPL-3.0-or-later	yes, dynamically linked	default
expo-media-library, expo-video	Mobile library index and preview	MIT	yes	default on mobile
PhotoKit	Apple Photos access in the desktop helper	Apple system framework	yes, on macOS	default on macOS

Sidecar libraries

FastAPI (MIT), Uvicorn (BSD-3-Clause), Pydantic (MIT), NumPy (BSD-3-Clause), OpenCV opencv-python-headless (Apache-2.0, wheel bundles LGPL FFmpeg), soundfile (BSD-3-Clause), PyTorch and torchvision (BSD-3-Clause), ONNX Runtime (MIT), faster-whisper (MIT), CTranslate2 (MIT), Hugging Face tokenizers (Apache-2.0), sam2 (Apache-2.0), Hydra and iopath (MIT). All are commercially usable.

Remotion

The spec proposed Remotion for motion graphics. It is not used: Remotion requires a paid company license above a team-size threshold and needs headless Chromium. The motion renderer instead draws each template as an SVG per frame, rasterizes it with resvg and encodes a QuickTime Animation (alpha) clip with ffmpeg, behind the same MotionGraphicsRenderer interface.

The license entries above were taken from the upstream license files as recorded in infra/media-worker/LICENSES.md, the pinned registry in infra/media-worker/app/models.py and the installed npm package metadata. The upstream pages themselves could not be fetched again while this document was written; re-check them before relying on this table for a release.

Known limitations

• Content classifier is experimental. It is zero-shot SigLIP with text prompts, not a trained moderation model. Treat its scores as a first filter; it does not replace a platform's own review. Segmentation and inpainting are experimental too, and inpainting with a low quality score falls back to a cover or reframe suggestion.
• Synthetic identity is unavailable. No commercially licensed model was found, so the mode reports "not available".
• CUDA and ROCm images are unbuilt in CI. Only the CPU image is built and tested; the GPU variants are defined but untested.
• Job limits are per worker process. MEDIA_GPU_CONCURRENCY and MEDIA_CPU_CONCURRENCY bound one worker; several workers multiply them. The sidecar's own gates (MEDIA_GPU_CONCURRENCY, MEDIA_QUEUE_SIZE) still cap inference and answer busy when full.
• Motion graphics use resvg, not Remotion (see above): templates are SVG, so effects needing a browser engine (CSS, WebGL) are not available.
• Scheduled publishing is a media.publish background job with a run time, not a Routine. Rescheduling replaces the job and cancelling removes it.
• Platform profiles are unverified until someone checks them against the official pages listed in each file.
• Own site output is a progressive MP4; no HLS ladder is produced.
• Apple Photos needs the macOS desktop app; the source is not offered on Windows, Linux or the web.