Compare commits

...

59 Commits

Author SHA1 Message Date
Claude
26872b5dc4
test(media): library e2e covering sessions, people consent, results import and 50k virtualization 2026-10-03 19:29:21 +00:00
Claude
80ca23c546
feat(media): collection, people with consent capture, and results views 2026-10-03 19:18:50 +00:00
Claude
9a3423e458
feat(media): library view with virtualized grid, filters, sources and studio chrome 2026-10-03 19:16:05 +00:00
Claude
159fafdb55
feat(media): shared web helpers for byte routes, polling and studio context 2026-10-03 19:10:10 +00:00
Claude
884d8903ca
feat(media): web view skeleton for Media Studio
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:04:47 +00:00
Claude
b4083844ce
feat(media): wave 2 integration
Composition root shared by API and worker, media job handlers on every job
host, HTTP byte routes, consent re-check before bundling, and auto-fix limited
to changes that keep protection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:04:01 +00:00
Claude
90f18cea52
merge: W2 privacy 2026-10-03 18:57:49 +00:00
Claude
d88e0de1bc
wip: media composition root 2026-10-03 18:57:49 +00:00
Claude
8d07ab17d4
feat(media): anonymization, marks, consent, and the render gate
prepareSources applies every face, mark, and region protection before the
compiler runs and fails closed when one cannot be honored: all face modes,
tracking with interpolation and grown holds over lost detections, reframe_out
crops with blur fallback, synthetic identity only through a licensed provider
(blur fallback otherwise), inpainting with a quality floor that falls back to
a cover, and person-bound marks re-found next to the face. The gate runs on
the render output: classifier at the profile rate plus cuts, face tracking
matched to people locally, pixel verification of every recorded protection,
consent and age proof (always for paid video), text and technical checks.
Auto-fix proposes only compliance repairs and stops after three rounds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:57:04 +00:00
Claude
2973f5074b
merge: W2 ingest
# Conflicts:
#	apps/desktop/src/preload.test.ts
2026-10-03 18:51:13 +00:00
Claude
21edc4add7
feat(media): wire marketing into the API, add segment results and teaser reset
Brand kits are validated through saveBrandKit and return their warnings;
manual metrics refit the teaser model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:50:45 +00:00
Claude
b416ff0fdf
feat(media): ingest job handlers with bounded concurrency and disk backpressure
media.sync, media.proxy, media.analyze, and media.cluster for Media Studio:
incremental metadata sync with deletion detection, 720p CFR proxies with
1 s keyframes, tone mapping, posters and sprites, sidecar analyses stored per
model version with scored segments, and sessions, themes, and person
suggestions. Proxy and analysis work is pumped in a small per-space window,
jobs step aside when their GPU or CPU permit is taken, and a full disk or an
unreachable sidecar pauses work instead of failing assets. Re-clustering
keeps collections the user merged or split. Fixes photo posters, which
never rendered because ffmpeg yields no frame when seeking a single image.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:50:30 +00:00
Claude
f29e6379cb
merge: W2 worktree-agent-a78be97f0347925a3 2026-10-03 18:47:30 +00:00
Claude
0e82982452
merge: W2 media-w2-render 2026-10-03 18:47:29 +00:00
Claude
227703a78a
merge: W2 worktree-agent-a346544dc365ebb55 2026-10-03 18:47:29 +00:00
Claude
4876c58fca
style(media): format zip test 2026-10-03 18:46:41 +00:00
Claude
7c744b98a0
feat(media): per-segment results and copy checks on every field 2026-10-03 18:46:36 +00:00
Claude
3ebe0fe47b
feat(media): ffmpeg renderer, render job and real render integration test
Fail closed when anonymization cannot prepare every clip, retry a failing
hardware encoder in software, verify each ffmpeg output exists, and fail
renders whose video or poster still carries source metadata. Cover parsing,
encoder detection, two-pass loudness, abort and cleanup, the render job
(auto-fix, cancel, outputs) on memory and Postgres, and a synthetic
16:9 to 9:16 render plus a region blur check with ffprobe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:44:58 +00:00
Claude
03e4961c9f
test(media): cover marketing plans, brand kits, bundles, publishing and learning 2026-10-03 18:43:37 +00:00
Claude
ffcc47710d
test(media): type the media router test client 2026-10-03 18:39:28 +00:00
Claude
e601b26292
feat(media): media API procedures, chunked uploads, and authorization tests 2026-10-03 18:37:51 +00:00
Claude
872fdd2194
feat(media): marketing service, brand kits, intro variants and bundle publisher 2026-10-03 18:36:53 +00:00
Claude
87d5be4daf
fix(desktop): iterate Photos persistent changes as a sequence
PHPersistentChangeFetchResult has no enumerateChanges; it is a Swift sequence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:31:45 +00:00
Claude
44396969fc
merge: main into media studio
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:30:25 +00:00
Claude
393137be9a
Merge remote-tracking branch 'origin/media-wip/w2-render' into media-w2-render 2026-10-03 18:29:04 +00:00
Claude
4721f00546
Merge remote-tracking branch 'origin/media-wip/w2-api' into worktree-agent-a346544dc365ebb55 2026-10-03 18:28:41 +00:00
Claude
d6957c4670
Merge remote-tracking branch 'origin/media-wip/w2-privacy' into worktree-agent-ac408b7c790dd9920 2026-10-03 18:28:38 +00:00
Claude
439de0d936
merge: media base b4222ac 2026-10-03 18:28:20 +00:00
Claude
3c13961948
Merge remote-tracking branch 'origin/media-wip/w2-marketing' into worktree-agent-a78be97f0347925a3 2026-10-03 18:28:15 +00:00
Claude
f4bf545c45
Merge remote-tracking branch 'origin/media-wip/w2-ingest' into worktree-agent-afca57be83bd9f7ba 2026-10-03 18:28:03 +00:00
Claude
7f6fc678c3
wip(media): w2-privacy partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:55 +00:00
Claude
3f9a601dba
wip(media): w2-marketing partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:53 +00:00
Claude
8f38c22119
wip(media): w2-render partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:51 +00:00
Claude
239a1eceb0
wip(media): w2-api partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:49 +00:00
Claude
55e129cb7e
wip(media): w2-ingest partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:47 +00:00
Claude
b4222ac5ec
feat(media): compose profile and CI job for the media sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:21:16 +00:00
Claude
bd5f637501
merge: W1-sidecar local media inference sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:20:41 +00:00
Claude
5e2094f2ad
feat(media): local inference sidecar for Media Studio
FastAPI service in infra/media-worker implementing every
MediaInferenceProvider operation behind HTTP API v1, plus capabilities.
Bearer auth, MEDIA_ROOT realpath enforcement, strict request schemas,
cancellation on disconnect, GPU/CPU gates with 503 overflow, input
size/duration limits, lazy SHA256-pinned model downloads, fail-closed
redaction with atomic output, offline pytest suite with fake backends,
CPU/CUDA/ROCm Dockerfile and a license review of every model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:19:47 +00:00
Claude
a40582165a
feat(media): wire Apple Photos sync into the desktop app
Move the Photos bridge types into the desktop contract, expose
rakazoDesktop.media.photos from the preload, connect the sync runner in the
main process through an authenticated oRPC client, package and sign the
helper, and build and test it on macOS CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:15:53 +00:00
Claude
2556ed197e
feat(media): service contracts between media slices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:11:04 +00:00
Claude
475710e27e
feat(media): shared service deps and consent-gated model client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:10:02 +00:00
Claude
d43e374534
merge: W1-domain media domain logic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:39 +00:00
Claude
a6598f09d9
test(media): cover clustering, EDL, compiler, teaser, gate and marketing
Includes compiler snapshots, injection-proof escaping checks, and an
ffmpeg integration test on synthetic testsrc/sine inputs that is skipped
when ffmpeg is not installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
26d95a374e
feat(media): implement pure domain logic for Media Studio
Session and theme clustering, EDL validation and timeline mapping, the
EDL-to-ffmpeg compiler with caption files and filtergraph escaping,
teaser selection and learning, the compliance gate with compliance-only
auto-fix, and the marketing helpers (UTM, hashtags, CSV metrics, intro
proposals, brand-kit checks, upload checklist).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
1aab5ab3a8
feat(media): expose asset externalId, filter by ingest state, push device deletions
The desktop Photos sync needs to map awaiting assets back to Photos items,
query only awaiting uploads, and report deletions without a full relisting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:00:02 +00:00
Claude
51445e9442
merge: W1-desktop Apple Photos helper and sync
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:20 +00:00
Claude
0ee6c664d2
build(db): depend on @rakazo/media for coordinate rounding
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:02 +00:00
Claude
c877343218
merge: W1-db media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:32 +00:00
Claude
d3c6f262ad
feat(media): Apple Photos helper and client-pushed desktop sync
Adds the macOS photos-helper Swift CLI (PhotoKit listing with stable
cursors, persistent change tokens, iCloud-aware original export with a
path guard and SHA-256) plus XCTests for its PhotoKit-free core, and the
Electron main-process side: a validated helper client, a resumable
metadata push and chunked original upload against the media contract,
and IPC handlers that stay unavailable off macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:14 +00:00
Claude
5e155a0850
test(media): Postgres and unit tests for the media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c1c39b6e46
feat(media): space-scoped Media Studio repositories
Sources, assets with keyset search, analyses, segments, uploads,
collections with re-clustering, people with consent and complete
biometric removal, marks, brand kits, projects with versioned plans,
renders with atomic claims, publications, metrics and the funnel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c3f06e4a12
feat(media): wire media adapters and ffmpeg for CI media tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:53:43 +00:00
Claude
0c4e0202f3
merge: W1-adapters emulator, sidecar client, folder source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:59 +00:00
Claude
270ea8068a
feat(media): inference emulator, sidecar client, folder source and factory
Add the Media Studio adapters behind the frozen provider contracts: a
deterministic offline inference emulator driven by file-name markers that
renders real redactions with ffmpeg, a synthetic emulator library, the
bearer-auth HTTP client for the local media sidecar with validated
responses, typed errors, busy retries and timeouts, a lazily paginated
folder source that never follows symlinks, and the configuration factory.
A shared conformance suite runs against the emulator and, in the Docker
integration job, against the sidecar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:17 +00:00
Claude
f98fd9703c
feat(media): export motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:48:29 +00:00
Claude
759862fef4
merge: W1-motion motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:57 +00:00
Claude
b5cef21f44
feat(media): resvg motion graphics renderer with brand-kit templates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:24 +00:00
Claude
7ccc96ac11
feat(media): shared geo, path-safety and profile loading helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:34:04 +00:00
Claude
a663764ee8
feat(media): contracts and foundation for Media Studio
Freeze the shared contracts every Media Studio slice builds on: zod DTOs,
EDL, platform profile, brand kit, gate and render plan schemas; the media
RPC surface with a scaffold router; provider interfaces for sources,
inference, rendering, motion graphics and publishing; media background
jobs; the Prisma models and migration; and the @rakazo/media domain
package signatures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:32:52 +00:00
257 changed files with 55357 additions and 6 deletions

View File

@ -236,3 +236,15 @@ EVOO_COCKPIT_CORS_ORIGIN=
# Expo: English or Simplified Chinese (en | zh-CN). Overridden by SecureStore key
# rakazo.uiLocale from Account → Language.
# EXPO_PUBLIC_DEFAULT_UI_LOCALE=en
# Optional Media Studio worker (local ffmpeg + ML sidecar, compose profile "media").
# Without it, Media Studio stays hidden and media jobs are skipped.
# Compose: http://media-worker:8790. Bare-metal dev: http://127.0.0.1:8790.
# MEDIA_WORKER_URL=http://media-worker:8790
# At least 32 random characters.
# MEDIA_WORKER_TOKEN=REPLACE_WITH_RANDOM_TOKEN
# Private platform profiles that override the shipped ones. Keep outside the repo.
# MEDIA_PROFILES_DIR=~/.evoo/media-profiles
# Media jobs per worker process: model inference (GPU) and ffmpeg work (CPU).
# MEDIA_GPU_CONCURRENCY=1
# MEDIA_CPU_CONCURRENCY=2

View File

@ -170,6 +170,15 @@ jobs:
- run: pnpm db:generate
env:
DATABASE_URL: postgres://rakazo:rakazo@127.0.0.1:5433/rakazo
- name: Ensure ffmpeg for media tests
shell: bash
run: |
if command -v ffmpeg >/dev/null && command -v ffprobe >/dev/null; then exit 0; fi
if sudo -n true 2>/dev/null; then
sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
else
echo "::warning::ffmpeg is missing on this runner; media render tests will be skipped"
fi
- run: pnpm test
test-integration:

View File

@ -0,0 +1,32 @@
name: desktop-photos-helper
on:
pull_request:
paths: &photos-helper-paths
- ".github/workflows/desktop-photos-helper.yml"
- "apps/desktop/native/photos-helper/**"
push:
branches: [main]
paths: *photos-helper-paths
permissions:
contents: read
concurrency:
group: desktop-photos-helper-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
swift:
name: Photos helper build and XCTest
runs-on: [self-hosted, evoo-ci, macOS]
timeout-minutes: 20
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Build and test
working-directory: apps/desktop/native/photos-helper
run: |
set -euo pipefail
swift build -c release
swift test

32
.github/workflows/media-worker.yml vendored Normal file
View File

@ -0,0 +1,32 @@
name: media-worker
on:
pull_request:
paths: &media-worker-paths
- ".github/workflows/media-worker.yml"
- "infra/media-worker/**"
push:
branches: [main]
paths: *media-worker-paths
permissions:
contents: read
concurrency:
group: media-worker-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
sidecar:
name: Media sidecar
runs-on: [self-hosted, evoo-ci, Linux]
timeout-minutes: 30
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Build test image (CPU)
run: docker build --build-arg ACCELERATOR=cpu --target test -t rakazo/media-worker:ci-test infra/media-worker
- name: Pytest (offline, fake model backends)
run: docker run --rm --network none rakazo/media-worker:ci-test
- name: Build runtime image (CPU)
run: docker build --build-arg ACCELERATOR=cpu -t rakazo/media-worker:ci infra/media-worker

View File

@ -145,6 +145,14 @@ jobs:
umask 077
printf '%s\n' "$APPLE_API_KEY_P8" > "$APPLE_API_KEY"
chmod 600 "$APPLE_API_KEY"
- name: Build and test the universal Photos helper
if: runner.os == 'macOS'
working-directory: apps/desktop/native/photos-helper
run: |
set -euo pipefail
swift test
swift build -c release --arch arm64 --arch x86_64
lipo -verify_arch .build/apple/Products/Release/photos-helper arm64 x86_64
- name: Package signed and notarized universal macOS app
if: runner.os == 'macOS'
env:
@ -174,6 +182,10 @@ jobs:
grep -Fqx "provider: github" "$app/Contents/Resources/app-update.yml"
grep -Fqx "owner: elie222" "$app/Contents/Resources/app-update.yml"
grep -Fqx "repo: rakazo" "$app/Contents/Resources/app-update.yml"
helper="$app/Contents/Resources/bin/photos-helper"
codesign --verify --strict --verbose=2 "$helper"
codesign -d --entitlements - --xml "$helper" | grep -Fq "com.apple.security.personal-information.photos-library"
/usr/libexec/PlistBuddy -c 'Print :NSPhotoLibraryUsageDescription' "$app/Contents/Info.plist"
- name: Require Windows signing credentials
if: runner.os == 'Windows'

View File

@ -19,6 +19,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/db": "workspace:*",
"@rakazo/logging": "workspace:*",
"@rakazo/media": "workspace:*",
"@rakazo/memory": "workspace:*",
"hono": "4.13.7"
},

View File

@ -25,6 +25,8 @@ import {
createCloudAgentConnection,
createConnectorStack,
createJobReconciler,
createMediaJobHandlers,
createMediaServices,
createMessagingContextLoader,
createMessagingTeamChatSender,
createProfileConsolidationScheduler,
@ -92,6 +94,7 @@ import { cors } from "hono/cors";
import type { AppEnv } from "./env.js";
import { loadEnv } from "./env.js";
import { mountLocalSettings } from "./local-settings.js";
import { mountMediaHttpRoutes } from "./media-http.js";
import {
createMessagingInboundHandler,
teamChatSenderCanWakeMessageRoutines,
@ -414,6 +417,18 @@ export async function createApp(
shutdownSignal: shutdown.signal,
});
const mediaServices = createMediaServices({
config: env.media,
prisma,
jobs,
artifacts,
runtime,
realtime,
resolveModel: (scope) => executor.resolveModel(scope),
});
const mediaJobs = mediaServices
? createMediaJobHandlers(mediaServices, { signal: shutdown.signal })
: null;
const jobHandlers = createBackgroundJobHandlers({
executor,
prisma,
@ -421,6 +436,7 @@ export async function createApp(
home,
jobs,
events,
...(mediaJobs ? { media: mediaJobs.handlers } : {}),
workerId: "api",
runtime,
secretStore: secrets,
@ -438,6 +454,11 @@ export async function createApp(
});
await graphileWorkerHost.start(jobHandlers);
}
if (mediaJobs) {
void mediaJobs
.resume()
.catch((error) => getLogger().error("media ingest resume failed", error));
}
const reconciler = inMemoryJobs
? createJobReconciler({
prisma,
@ -469,6 +490,7 @@ export async function createApp(
remoteConnectors,
artifacts,
dataDir: env.dataDir,
mediaServices,
messaging: {
enabled: Boolean(messaging),
providers: messaging?.platforms().map((platform) => platform.provider) ?? [],
@ -583,6 +605,13 @@ export async function createApp(
if (actor) enrichLogContext({ "user.id": actor.userId, "space.id": actor.spaceId });
return actor;
});
mountMediaHttpRoutes(app, { services: mediaServices }, async (c) => {
const session = await auth.api.getSession({ headers: sessionHeaders(c.req.raw) });
if (!session?.user) return null;
return requireMembership(prisma, session.user.id, c.req.header("x-rakazo-space-id")).catch(
() => null,
);
});
mountWebhookHttpRoutes(app, { prisma, secrets, events, jobs });
// Shared with stop so a shutdown during retry delays does not restart polling.
let messagingStopped = false;

View File

@ -1,4 +1,6 @@
import type { MediaConfig } from "@rakazo/adapters";
import {
mediaConfigFromEnv,
resolveCloudAgentProvider,
resolveDeploymentModel,
resolveSandboxProvider,
@ -15,6 +17,8 @@ export { resolveCloudAgentProvider, resolveSandboxProvider } from "@rakazo/adapt
export interface AppEnv {
nodeEnv: string;
/** Media Studio; disabled unless a sidecar (or the scripted emulator) is configured. */
media: MediaConfig;
desktopStackToken?: string;
databaseUrl: string;
realtimeDatabaseUrl: string;
@ -106,6 +110,7 @@ export function loadEnv(source: NodeJS.ProcessEnv = process.env): AppEnv {
const updaterToken = optional(source.RAKAZO_UPDATER_TOKEN);
return {
nodeEnv: source.NODE_ENV ?? "",
media: mediaConfigFromEnv(source),
databaseUrl: required(source, "DATABASE_URL"),
realtimeDatabaseUrl: source.REALTIME_DATABASE_URL ?? required(source, "DATABASE_URL"),
desktopStackToken: optional(source.RAKAZO_DESKTOP_STACK_TOKEN),

265
apps/api/src/media-http.ts Normal file
View File

@ -0,0 +1,265 @@
import { createReadStream } from "node:fs";
import { mkdir, stat } from "node:fs/promises";
import path from "node:path";
import { Readable } from "node:stream";
import type { MediaServices } from "@rakazo/adapters";
import type { Actor } from "@rakazo/contracts";
import { IsolationError, MediaStateConflictError } from "@rakazo/db";
import { assertInside, PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import type { Context, Hono } from "hono";
import { cancelBody } from "./http-body.js";
import { MEDIA_CHUNK_TIMEOUT_MS, UploadChunkError, writeUploadChunk } from "./media-uploads.js";
/*
* Byte routes Media Studio cannot express as JSON RPC: chunk uploads and
* streaming of proxies, thumbnails, renders, and bundles. Every id is checked
* against the actor's space before a path is derived, and every path is
* resolved inside the media root.
*/
const ID = /^[A-Za-z0-9_-]{1,64}$/;
const FILE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/;
const CHUNK_INDEX = /^[0-9]{1,7}$/;
const CONTENT_TYPES: Record<string, string> = {
".mp4": "video/mp4",
".m4v": "video/mp4",
".mov": "video/quicktime",
".webm": "video/webm",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".png": "image/png",
".webp": "image/webp",
".gif": "image/gif",
".srt": "application/x-subrip",
".vtt": "text/vtt; charset=utf-8",
".json": "application/json",
".md": "text/markdown; charset=utf-8",
".txt": "text/plain; charset=utf-8",
".zip": "application/zip",
".m4a": "audio/mp4",
".wav": "audio/wav",
};
export function mediaContentType(fileName: string): string {
return CONTENT_TYPES[path.extname(fileName).toLowerCase()] ?? "application/octet-stream";
}
export type ByteRange = { start: number; end: number };
/**
* Parse a single `bytes=` range. Returns null for no (or an ignorable
* multi-part) range and "unsatisfiable" when it cannot be served.
*/
export function parseByteRange(
header: string | undefined,
size: number,
): ByteRange | null | "unsatisfiable" {
if (!header) return null;
const match = /^bytes=(\d*)-(\d*)$/.exec(header.trim());
if (!match) return header.includes(",") ? null : "unsatisfiable";
const [, first = "", last = ""] = match;
if (first === "" && last === "") return "unsatisfiable";
if (first === "") {
const suffix = Number(last);
if (suffix === 0 || size === 0) return "unsatisfiable";
return { start: Math.max(0, size - suffix), end: size - 1 };
}
const start = Number(first);
const end = last === "" ? size - 1 : Math.min(Number(last), size - 1);
if (start >= size || end < start) return "unsatisfiable";
return { start, end };
}
async function sendFile(
c: Context,
absolutePath: string,
options: { fileName: string; download?: boolean },
): Promise<Response> {
const info = await stat(absolutePath).catch(() => null);
if (!info?.isFile()) return c.json({ error: "Not found" }, 404);
const size = info.size;
const headers: Record<string, string> = {
"content-type": mediaContentType(options.fileName),
"accept-ranges": "bytes",
"cache-control": "private, max-age=300",
};
if (options.download) {
headers["content-disposition"] = `attachment; filename="${options.fileName}"`;
}
const range = parseByteRange(c.req.header("range"), size);
if (range === "unsatisfiable") {
return new Response(null, {
status: 416,
headers: { ...headers, "content-range": `bytes */${size}` },
});
}
if (c.req.method === "HEAD") {
return new Response(null, {
status: 200,
headers: { ...headers, "content-length": String(size) },
});
}
const start = range?.start ?? 0;
const end = range?.end ?? size - 1;
const body =
size === 0
? null
: (Readable.toWeb(
createReadStream(absolutePath, { start, end }),
) as ReadableStream<Uint8Array>);
return new Response(body, {
status: range ? 206 : 200,
headers: {
...headers,
"content-length": String(size === 0 ? 0 : end - start + 1),
...(range ? { "content-range": `bytes ${start}-${end}/${size}` } : {}),
},
});
}
function mediaHttpError(c: Context, error: unknown): Response {
if (error instanceof IsolationError || error instanceof PathOutsideRootError) {
return c.json({ error: "Not found" }, 404);
}
if (error instanceof UploadChunkError) return c.json({ error: error.message }, error.status);
if (error instanceof MediaStateConflictError) return c.json({ error: error.message }, 409);
if ((error as NodeJS.ErrnoException | null)?.code === "ENOENT") {
return c.json({ error: "Not found" }, 404);
}
throw error;
}
export function mountMediaHttpRoutes(
app: Hono,
deps: {
/** Null when Media Studio is not configured: every route answers 404. */
services: MediaServices | null;
chunkTimeoutMs?: number;
},
resolveActor: (c: Context) => Promise<Actor | null>,
) {
async function authorize(
c: Context,
): Promise<
{ actor: Actor; services: MediaServices; response?: undefined } | { response: Response }
> {
const actor = await resolveActor(c);
if (!actor) return { response: c.json({ error: "Unauthorized" }, 401) };
if (!deps.services)
return { response: c.json({ error: "Media Studio is not configured" }, 404) };
return { actor, services: deps.services };
}
async function mediaFile(services: MediaServices, relative: string | null) {
if (!relative) return null;
await mkdir(services.deps.mediaRoot, { recursive: true });
return resolveInside(services.deps.mediaRoot, relative);
}
app.put("/api/media/uploads/:uploadId/chunks/:index", async (c) => {
const auth = await authorize(c);
if (auth.response) {
cancelBody(c.req.raw.body);
return auth.response;
}
const uploadId = c.req.param("uploadId");
const index = c.req.param("index");
if (!ID.test(uploadId) || !CHUNK_INDEX.test(index)) {
cancelBody(c.req.raw.body);
return c.json({ error: "Not found" }, 404);
}
const lengthHeader = c.req.header("content-length");
const declared = lengthHeader && /^[0-9]+$/.test(lengthHeader) ? Number(lengthHeader) : null;
const timeout = AbortSignal.timeout(deps.chunkTimeoutMs ?? MEDIA_CHUNK_TIMEOUT_MS);
const body = c.req.raw.body;
try {
const upload = await writeUploadChunk({
services: auth.services,
spaceId: auth.actor.spaceId,
uploadId,
index: Number(index),
body,
declaredLength: declared,
signal: AbortSignal.any([timeout, c.req.raw.signal]),
});
return c.json(upload);
} catch (error) {
cancelBody(body);
if (timeout.aborted) return c.json({ error: "Chunk took too long" }, 408);
return mediaHttpError(c, error);
}
});
app.get("/api/media/renders/:renderId/files/:fileName", async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const renderId = c.req.param("renderId");
const fileName = c.req.param("fileName");
if (!ID.test(renderId) || !FILE_NAME.test(fileName) || fileName.includes("..")) {
return c.json({ error: "Not found" }, 404);
}
try {
const render = await auth.services.deps.repos.getRenderRecord(auth.actor.spaceId, renderId);
const download = c.req.query("download") === "1";
// Only a render that passed its gate leaves the app as a download.
if (download && render.status !== "done") return c.json({ error: "Not found" }, 404);
const absolute = await auth.services.render.outputPath(auth.actor, renderId, fileName);
if (!absolute) return c.json({ error: "Not found" }, 404);
await mkdir(auth.services.deps.mediaRoot, { recursive: true });
const inside = await assertInside(
await resolveInside(
auth.services.deps.mediaRoot,
`${auth.actor.spaceId}/renders/${renderId}`,
),
absolute,
);
return await sendFile(c, inside, { fileName, download });
} catch (error) {
return mediaHttpError(c, error);
}
});
for (const kind of ["thumb", "proxy"] as const) {
app.get(`/api/media/assets/:assetId/${kind}`, async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const assetId = c.req.param("assetId");
if (!ID.test(assetId)) return c.json({ error: "Not found" }, 404);
try {
const asset = await auth.services.deps.repos.getAssetRecord(auth.actor.spaceId, assetId);
const relative = kind === "thumb" ? asset.thumbPath : asset.proxyPath;
const absolute = await mediaFile(auth.services, relative);
if (!absolute) return c.json({ error: "Not found" }, 404);
return await sendFile(c, absolute, { fileName: path.basename(absolute) });
} catch (error) {
return mediaHttpError(c, error);
}
});
}
app.get("/api/media/bundles/:publicationId", async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const publicationId = c.req.param("publicationId");
if (!ID.test(publicationId)) return c.json({ error: "Not found" }, 404);
try {
const publication = await auth.services.deps.repos.getPublicationRecord(
auth.actor.spaceId,
publicationId,
);
if (publication.bundleArtifactId) return c.json({ error: "Not found" }, 404);
const absolute = await mediaFile(
auth.services,
`${auth.actor.spaceId}/bundles/${publication.id}/bundle.zip`,
);
if (!absolute) return c.json({ error: "Not found" }, 404);
return await sendFile(c, absolute, {
fileName: `bundle-${publication.id}.zip`,
download: true,
});
} catch (error) {
return mediaHttpError(c, error);
}
});
}

View File

@ -0,0 +1,389 @@
import type { MediaInferenceProvider } from "@rakazo/adapter-kit";
import type {
MediaActor,
MediaIngestService,
MediaMarketingService,
MediaPrivacyService,
MediaRenderService,
MediaServiceDeps,
MediaServices,
} from "@rakazo/adapters";
import type { PlatformProfile } from "@rakazo/contracts";
import { mediaRpcShapes } from "@rakazo/contracts";
import type { MediaRepos, PrismaClient } from "@rakazo/db";
/*
* In-memory stand-ins for the media service slices, for API tests only. They
* record what the router hands them and do the minimum through the real
* repositories; nothing here renders, detects, or publishes.
*/
export interface MediaServiceCall {
service: "ingest" | "privacy" | "render" | "marketing";
method: string;
actor: MediaActor;
input: unknown;
}
export interface FakeMediaServicesOptions {
repos: MediaRepos;
mediaRoot: string;
prisma?: PrismaClient;
inference?: MediaInferenceProvider | null;
profiles?: PlatformProfile[];
now?: () => Date;
/** Render id → absolute output path the fake render service reports. */
renderOutputs?: Map<string, string>;
onUploadCompleted?: (
actor: MediaActor,
input: { assetId: string; tempPath: string; sha256: string; fileName: string },
) => Promise<void>;
}
export function fakeMediaServices(options: FakeMediaServicesOptions): {
services: MediaServices;
calls: MediaServiceCall[];
} {
const calls: MediaServiceCall[] = [];
const { repos } = options;
const record = (
service: MediaServiceCall["service"],
method: string,
actor: MediaActor,
input: unknown,
) => {
calls.push({ service, method, actor: { spaceId: actor.spaceId, userId: actor.userId }, input });
};
const ingest: MediaIngestService = {
async pushItems(actor, input) {
record("ingest", "pushItems", actor, input);
return { created: 0, updated: 0, unchanged: 0, deleted: 0 };
},
async requestSync(actor, sourceId) {
record("ingest", "requestSync", actor, sourceId);
},
async uploadCompleted(actor, input) {
record("ingest", "uploadCompleted", actor, input);
await options.onUploadCompleted?.(actor, input);
},
async recluster(actor) {
record("ingest", "recluster", actor, null);
},
async removeSource(actor, sourceId) {
record("ingest", "removeSource", actor, sourceId);
await repos.removeSource(actor.spaceId, sourceId);
},
};
const privacy: MediaPrivacyService = {
async prepareSources() {
throw new Error("prepareSources is not part of the API fakes");
},
async gate() {
throw new Error("gate is not part of the API fakes");
},
async createMark(actor, input) {
record("privacy", "createMark", actor, input);
return repos.createMark(actor.spaceId, input);
},
async updateMark(actor, input) {
record("privacy", "updateMark", actor, input);
const { markId, ...patch } = input;
return repos.updateMark(actor.spaceId, markId, patch);
},
async removePerson(actor, personId) {
record("privacy", "removePerson", actor, personId);
await repos.removePersonCompletely(actor.spaceId, personId);
},
};
const render: MediaRenderService = {
async start(actor, input) {
record("render", "start", actor, input);
return repos.createRender(actor.spaceId, { ...input, createdBy: actor.userId });
},
async cancel(actor, renderId) {
record("render", "cancel", actor, renderId);
return repos.cancelRender(actor.spaceId, renderId);
},
async outputPath(actor, renderId, fileName) {
record("render", "outputPath", actor, { renderId, fileName });
const file = options.renderOutputs?.get(renderId);
return file?.endsWith(`/${fileName}`) ? file : null;
},
};
const marketing: MediaMarketingService = {
async autoPlan(actor, input) {
record("marketing", "autoPlan", actor, input);
return [];
},
async submitPlan(actor, input) {
record("marketing", "submitPlan", actor, input);
return { ok: false, errors: [] };
},
async introVariants(actor, input) {
record("marketing", "introVariants", actor, input);
return [];
},
async thumbnails(actor, input) {
record("marketing", "thumbnails", actor, input);
return [];
},
async copy(actor, input) {
record("marketing", "copy", actor, input);
const render = await repos.getRender(actor.spaceId, input.renderId);
return {
profileId: render.profileId,
language: input.language,
title: "Title",
hook: "Hook",
caption: "Caption",
hashtags: [],
issues: [],
};
},
async createPublication(actor, input) {
record("marketing", "createPublication", actor, input);
return repos.createPublication(actor.spaceId, actor.userId, {
...input,
trackingUrl: null,
scheduledAt: input.scheduledAt ? new Date(input.scheduledAt) : null,
});
},
async bundle(actor, publicationId) {
record("marketing", "bundle", actor, publicationId);
return repos.markPublicationBundled(actor.spaceId, publicationId, null);
},
async cancelPublication(actor, publicationId) {
record("marketing", "cancelPublication", actor, publicationId);
return repos.cancelPublication(actor.spaceId, publicationId);
},
async importMetricsCsv(actor, input) {
record("marketing", "importMetricsCsv", actor, input);
return { imported: 0, skipped: [] };
},
};
const deps = {
prisma: options.prisma,
repos,
jobs: { publish: async () => undefined },
inference: options.inference ?? null,
renderer: {
id: "fake",
hardwareEncoders: async () => ["h264_fake"],
render: async () => {
throw new Error("render is not part of the API fakes");
},
},
motion: { id: "fake", templates: () => [], render: async () => ({ path: "", hasAlpha: true }) },
publisher: { id: "fake", publish: async () => ({ status: "bundled", externalUrl: null }) },
shortener: null,
sources: {},
mediaRoot: options.mediaRoot,
profiles: async () => ({
profiles: new Map(
(options.profiles ?? []).map((profile) => [
profile.id,
{ profile, origin: "builtin" as const },
]),
),
errors: [],
}),
artifacts: {},
model: { generate: async () => null },
publish: async () => undefined,
now: options.now ?? (() => new Date()),
} as unknown as MediaServiceDeps;
return { services: { deps, ingest, privacy, render, marketing }, calls };
}
/** Every RPC path of the media contract, e.g. "sources/list". */
export function mediaProcedurePaths(): string[] {
const paths: string[] = [];
const walk = (node: Record<string, unknown>, prefix: string[]) => {
for (const [key, value] of Object.entries(node)) {
const child = value as Record<string, unknown>;
if ("input" in child && "output" in child) paths.push([...prefix, key].join("/"));
else walk(child, [...prefix, key]);
}
};
walk(mediaRpcShapes, []);
return paths;
}
export interface MediaSampleIds {
sourceId: string;
assetId: string;
secondAssetId: string;
segmentId: string;
trackId: string;
personId: string;
markId: string;
collectionId: string;
brandKitId: string;
projectId: string;
planId: string;
renderId: string;
publicationId: string;
mappingId: string;
uploadId: string;
}
/**
* How a procedure must treat another space's ids: refuse them as not found,
* answer only from the caller's own space, or be covered by a dedicated test.
*/
export type ForeignExpectation = "not_found" | "scoped" | "dedicated";
/** A valid input for every media procedure, built from one space's ids. */
export function sampleMediaInputs(
ids: MediaSampleIds,
kit: unknown,
): Record<string, { input: unknown; foreign: ForeignExpectation }> {
const at = "2026-09-01T10:00:00.000Z";
return {
capabilities: { input: {}, foreign: "scoped" },
"sources/list": { input: {}, foreign: "scoped" },
"sources/createFolder": { input: { label: "Footage", path: "/" }, foreign: "dedicated" },
"sources/registerClient": {
input: { kind: "ios_library", label: "Phone", deviceId: "device-sample" },
foreign: "scoped",
},
"sources/pushItems": {
input: { sourceId: ids.sourceId, items: [], syncToken: null, complete: true },
foreign: "not_found",
},
"sources/sync": { input: { sourceId: ids.sourceId }, foreign: "not_found" },
"sources/remove": { input: { sourceId: ids.sourceId }, foreign: "not_found" },
"assets/search": { input: {}, foreign: "scoped" },
"assets/get": { input: { assetId: ids.assetId }, foreign: "not_found" },
"assets/setPaidOnly": {
input: { segmentIds: [ids.segmentId], paidOnly: true },
foreign: "not_found",
},
"assets/requestUpload": { input: { assetIds: [ids.assetId] }, foreign: "not_found" },
"uploads/begin": {
input: {
assetId: ids.secondAssetId,
sizeBytes: 10,
sha256: "a".repeat(64),
fileName: "clip.mp4",
},
foreign: "not_found",
},
"uploads/status": { input: { uploadId: ids.uploadId }, foreign: "not_found" },
"uploads/complete": { input: { uploadId: ids.uploadId }, foreign: "not_found" },
"collections/list": { input: {}, foreign: "scoped" },
"collections/get": { input: { collectionId: ids.collectionId }, foreign: "not_found" },
"collections/rename": {
input: { collectionId: ids.collectionId, title: "Renamed" },
foreign: "not_found",
},
"collections/merge": {
input: { collectionIds: [ids.collectionId, `${ids.collectionId}x`] },
foreign: "not_found",
},
"collections/split": {
input: { collectionId: ids.collectionId, atAssetId: ids.assetId },
foreign: "not_found",
},
"collections/createManual": {
input: { title: "Picks", assetIds: [ids.assetId] },
foreign: "not_found",
},
"collections/recluster": { input: {}, foreign: "scoped" },
"people/list": { input: {}, foreign: "scoped" },
"people/update": { input: { personId: ids.personId, displayName: "X" }, foreign: "not_found" },
"people/merge": {
input: { personIds: [ids.personId, `${ids.personId}x`] },
foreign: "not_found",
},
"people/assignTrack": { input: { trackId: ids.trackId, personId: null }, foreign: "not_found" },
"people/recordConsent": {
input: {
personId: ids.personId,
consentArtifactId: null,
consentRecordedAt: at,
ageVerificationMethod: "platform_verified",
ageVerifiedAt: at,
},
foreign: "not_found",
},
"people/remove": { input: { personId: ids.personId }, foreign: "not_found" },
"marks/list": { input: { assetId: ids.assetId }, foreign: "scoped" },
"marks/create": {
input: { kind: "tattoo", assetId: ids.assetId, prompt: { tMs: 0 }, treatment: "blur" },
foreign: "not_found",
},
"marks/update": { input: { markId: ids.markId, label: "x" }, foreign: "not_found" },
"marks/remove": { input: { markId: ids.markId }, foreign: "not_found" },
"brandKits/list": { input: {}, foreign: "scoped" },
"brandKits/upsert": { input: { id: ids.brandKitId, kit }, foreign: "not_found" },
"brandKits/remove": { input: { id: ids.brandKitId }, foreign: "not_found" },
"brandKits/templates": { input: {}, foreign: "scoped" },
"brandKits/introVariants": { input: { id: ids.brandKitId }, foreign: "not_found" },
"profiles/list": { input: {}, foreign: "scoped" },
"profiles/get": { input: { profileId: "example-short-vertical" }, foreign: "scoped" },
"projects/list": { input: {}, foreign: "scoped" },
"projects/get": { input: { projectId: ids.projectId }, foreign: "not_found" },
"projects/create": {
input: {
title: "Teaser",
goal: "teaser",
collectionIds: [ids.collectionId],
profileIds: ["example-short-vertical"],
},
foreign: "not_found",
},
"projects/update": { input: { projectId: ids.projectId, title: "X" }, foreign: "not_found" },
"projects/remove": { input: { projectId: ids.projectId }, foreign: "not_found" },
"projects/autoPlan": { input: { projectId: ids.projectId }, foreign: "not_found" },
"plans/submit": { input: { projectId: ids.projectId, edl: {} }, foreign: "not_found" },
"renders/start": { input: { planId: ids.planId, quality: "draft" }, foreign: "not_found" },
"renders/get": { input: { renderId: ids.renderId }, foreign: "not_found" },
"renders/cancel": { input: { renderId: ids.renderId }, foreign: "not_found" },
"renders/thumbnails": { input: { renderId: ids.renderId }, foreign: "not_found" },
"marketing/copy": { input: { renderId: ids.renderId }, foreign: "not_found" },
"publications/list": { input: { projectId: ids.projectId }, foreign: "scoped" },
"publications/create": { input: { renderId: ids.renderId }, foreign: "not_found" },
"publications/bundle": { input: { publicationId: ids.publicationId }, foreign: "not_found" },
"publications/markPublished": {
input: { publicationId: ids.publicationId, externalUrl: null },
foreign: "not_found",
},
"publications/cancel": { input: { publicationId: ids.publicationId }, foreign: "not_found" },
"metrics/mappings": { input: {}, foreign: "scoped" },
"metrics/saveMapping": {
input: {
id: ids.mappingId,
name: "Imported",
columns: { publication: "id", observedAt: "date" },
publicationKey: "id",
},
foreign: "not_found",
},
"metrics/importCsv": {
input: { mappingId: ids.mappingId, csv: "id,date" },
foreign: "not_found",
},
"metrics/record": {
input: {
publicationId: ids.publicationId,
observedAt: at,
views: 1,
clicks: 0,
conversions: 0,
revenueCents: 0,
currency: "EUR",
},
foreign: "not_found",
},
"metrics/funnel": { input: { projectId: ids.projectId }, foreign: "not_found" },
"metrics/segments": { input: { projectId: ids.projectId }, foreign: "not_found" },
"metrics/resetTeaserModel": { input: {}, foreign: "scoped" },
};
}

View File

@ -0,0 +1,244 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { mkdir, open, rename, rm, stat } from "node:fs/promises";
import path from "node:path";
import type { MediaActor, MediaServices } from "@rakazo/adapters";
import type { MediaUpload } from "@rakazo/contracts";
import type { MediaRepos } from "@rakazo/db";
import { MediaStateConflictError } from "@rakazo/db";
import { resolveInside } from "@rakazo/media/node";
/*
* Resumable chunked uploads. Each chunk lands in its own file
* (<spaceId>/uploads/<uploadId>/chunks/<index>.part, written to a temp name and
* renamed, so a repeated or concurrent PUT of the same index is harmless).
* Completion concatenates the chunks in order into <spaceId>/uploads/<uploadId>/file
* while hashing, verifies size and SHA-256, and hands the file to ingest.
*
* Limits: the contract caps the total size, every chunk must have its exact
* length, one chunk must arrive within MEDIA_CHUNK_TIMEOUT_MS, and an upload
* that saw no progress for MEDIA_UPLOAD_IDLE_MS expires and must start over.
*/
/** A single chunk must arrive within this window; slow links resume per chunk. */
export const MEDIA_CHUNK_TIMEOUT_MS = 10 * 60_000;
/** An open upload without a new chunk for this long is failed and its bytes removed. */
export const MEDIA_UPLOAD_IDLE_MS = 24 * 60 * 60_000;
export class UploadChunkError extends Error {
constructor(
readonly status: 400 | 409 | 413,
message: string,
) {
super(message);
this.name = "UploadChunkError";
}
}
type UploadRow = Awaited<ReturnType<MediaRepos["getUploadRecord"]>>;
type UploadSize = Pick<UploadRow, "sizeBytes" | "chunkBytes">;
export function uploadChunkCount(row: UploadSize): number {
return Number((row.sizeBytes + BigInt(row.chunkBytes) - 1n) / BigInt(row.chunkBytes));
}
/** Exact byte length chunk `index` must have: every chunk is full except the last. */
export function expectedChunkBytes(row: UploadSize, index: number): number {
const count = uploadChunkCount(row);
if (index < count - 1) return row.chunkBytes;
return Number(row.sizeBytes - BigInt(row.chunkBytes) * BigInt(count - 1));
}
function uploadDir(row: Pick<UploadRow, "spaceId" | "id">): string {
return `${row.spaceId}/uploads/${row.id}`;
}
async function mediaPath(mediaRoot: string, relative: string): Promise<string> {
await mkdir(mediaRoot, { recursive: true });
return resolveInside(mediaRoot, relative);
}
async function chunksDir(mediaRoot: string, row: UploadRow): Promise<string> {
const dir = await mediaPath(mediaRoot, `${uploadDir(row)}/chunks`);
await mkdir(dir, { recursive: true });
return dir;
}
async function removeUploadFiles(mediaRoot: string, row: UploadRow): Promise<void> {
await rm(await mediaPath(mediaRoot, uploadDir(row)), { recursive: true, force: true });
}
function isIdle(services: MediaServices, row: UploadRow): boolean {
return services.deps.now().getTime() - row.updatedAt.getTime() > MEDIA_UPLOAD_IDLE_MS;
}
/** Fail an idle open upload and drop its bytes. Returns true when it expired. */
async function expireIfIdle(services: MediaServices, row: UploadRow): Promise<boolean> {
if (row.status !== "open" || !isIdle(services, row)) return false;
await services.deps.repos.failUpload(row.spaceId, row.id).catch((error: unknown) => {
// A concurrent request already closed it; the files still go.
if (!(error instanceof MediaStateConflictError)) throw error;
});
await removeUploadFiles(services.deps.mediaRoot, row);
return true;
}
/**
* Open an upload, or resume the open one for the same asset and content. An
* idle one expires first so the client starts over with a fresh upload.
*/
export async function beginMediaUpload(
services: MediaServices,
actor: MediaActor,
input: { assetId: string; sizeBytes: number; sha256: string; fileName: string },
): Promise<MediaUpload> {
const { repos } = services.deps;
const asset = await repos.getAssetRecord(actor.spaceId, input.assetId);
// Device items wait for their original; files read from a source never upload.
if (asset.storage !== "remote" && asset.ingestState !== "awaiting_upload") {
throw new MediaStateConflictError("This item does not need an upload");
}
const begin = () =>
repos.beginUpload(actor.spaceId, { ...input, assetId: asset.id, userId: actor.userId });
const upload = await begin();
const row = await repos.getUploadRecord(actor.spaceId, upload.id);
return (await expireIfIdle(services, row)) ? begin() : upload;
}
/**
* Stream one chunk body to disk without buffering more than one network read.
* Rejects bodies longer or shorter than the chunk's exact size.
*/
export async function writeUploadChunk(input: {
services: MediaServices;
spaceId: string;
uploadId: string;
index: number;
body: ReadableStream<Uint8Array> | null;
declaredLength: number | null;
/** Aborting stops reading the body; the chunk is then discarded as incomplete. */
signal?: AbortSignal;
}): Promise<MediaUpload> {
const { services } = input;
const { repos, mediaRoot } = services.deps;
const row = await repos.getUploadRecord(input.spaceId, input.uploadId);
if (await expireIfIdle(services, row)) throw new UploadChunkError(409, "Upload expired");
if (row.status !== "open") throw new UploadChunkError(409, "Upload is not open");
const count = uploadChunkCount(row);
if (!Number.isInteger(input.index) || input.index < 0 || input.index >= count) {
throw new UploadChunkError(400, "Chunk index out of range");
}
const expected = expectedChunkBytes(row, input.index);
if (input.declaredLength !== null && input.declaredLength > expected) {
throw new UploadChunkError(413, "Chunk is too large");
}
if (!input.body) throw new UploadChunkError(400, "Chunk is empty");
const dir = await chunksDir(mediaRoot, row);
const finalPath = path.join(dir, `${input.index}.part`);
const tempPath = path.join(dir, `${input.index}.${randomUUID()}.tmp`);
const handle = await open(tempPath, "wx");
let written = 0;
let ok = false;
const reader = input.body.getReader();
const stop = () => void reader.cancel().catch(() => undefined);
input.signal?.addEventListener("abort", stop, { once: true });
try {
if (input.signal?.aborted) stop();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
written += value.byteLength;
if (written > expected) {
void reader.cancel().catch(() => undefined);
throw new UploadChunkError(413, "Chunk is too large");
}
await handle.write(value);
}
if (written !== expected) throw new UploadChunkError(400, "Chunk is incomplete");
ok = true;
} finally {
input.signal?.removeEventListener("abort", stop);
reader.releaseLock();
await handle.close();
if (!ok) await rm(tempPath, { force: true });
}
await rename(tempPath, finalPath);
return repos.markChunkReceived(input.spaceId, input.uploadId, input.index);
}
const completing = new Set<string>();
/**
* Assemble a fully received upload, verify it, hand it to ingest, and close it.
* A checksum mismatch fails the upload so the client starts a fresh one.
*/
export async function completeMediaUpload(
services: MediaServices,
actor: MediaActor,
uploadId: string,
): Promise<MediaUpload> {
const { repos, mediaRoot } = services.deps;
const row = await repos.getUploadRecord(actor.spaceId, uploadId);
if (row.status === "complete") return repos.uploadStatus(actor.spaceId, uploadId);
if (await expireIfIdle(services, row)) throw new UploadChunkError(409, "Upload expired");
if (row.status !== "open") throw new MediaStateConflictError("Upload is not open");
const received = new Set(row.receivedChunks);
const count = uploadChunkCount(row);
for (let index = 0; index < count; index++) {
if (!received.has(index)) throw new MediaStateConflictError("Upload is missing chunks");
}
if (completing.has(row.id)) throw new MediaStateConflictError("Upload is already completing");
completing.add(row.id);
try {
const dir = await chunksDir(mediaRoot, row);
const relativeFile = `${uploadDir(row)}/file`;
const filePath = await mediaPath(mediaRoot, relativeFile);
const tempFile = await mediaPath(mediaRoot, `${uploadDir(row)}/file.${randomUUID()}.tmp`);
const hash = createHash("sha256");
let total = 0;
const out = await open(tempFile, "wx");
try {
for (let index = 0; index < count; index++) {
const chunkPath = path.join(dir, `${index}.part`);
const info = await stat(chunkPath).catch(() => null);
if (!info || info.size !== expectedChunkBytes(row, index)) {
throw new MediaStateConflictError("Upload is missing chunks");
}
for await (const buffer of createReadStream(chunkPath) as AsyncIterable<Buffer>) {
hash.update(buffer);
total += buffer.byteLength;
await out.write(buffer);
}
}
} catch (error) {
await out.close();
await rm(tempFile, { force: true });
throw error;
}
await out.close();
const sha256 = hash.digest("hex");
if (sha256 !== row.sha256 || BigInt(total) !== row.sizeBytes) {
await removeUploadFiles(mediaRoot, row);
await repos.failUpload(actor.spaceId, uploadId);
throw new UploadChunkError(400, "Upload does not match its checksum");
}
await rename(tempFile, filePath);
await rm(dir, { recursive: true, force: true });
await services.ingest.uploadCompleted(actor, {
assetId: row.assetId,
tempPath: filePath,
sha256,
fileName: row.fileName,
});
// Ingest normally moves the file into the asset folder and closes the
// upload itself; otherwise the assembled file stays as the original.
const after = await repos.getUploadRecord(actor.spaceId, uploadId);
if (after.status === "complete") {
await removeUploadFiles(mediaRoot, row);
return repos.uploadStatus(actor.spaceId, uploadId);
}
return repos.completeUpload(actor.spaceId, uploadId, { storagePath: relativeFile });
} finally {
completing.delete(row.id);
}
}

View File

@ -0,0 +1,516 @@
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createRouterClient, ORPCError } from "@orpc/server";
import type { Actor } from "@rakazo/contracts";
import type { MediaRepos } from "@rakazo/db";
import { createMediaRepos } from "@rakazo/db";
import { Hono } from "hono";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import type { MediaTestDb, SeededMediaSpace } from "../../../packages/db/src/media-fixtures.js";
import {
openMediaTestDb,
seedMediaSpace,
sourceItem,
testBrandKit,
} from "../../../packages/db/src/media-fixtures.js";
import { testProfile } from "../../../packages/media/src/test-fixtures.js";
import { createMediaRouter } from "./media.js";
import { mountMediaHttpRoutes } from "./media-http.js";
import type { MediaServiceCall } from "./media-test-fakes.js";
import { fakeMediaServices, sampleMediaInputs } from "./media-test-fakes.js";
import { MEDIA_UPLOAD_IDLE_MS } from "./media-uploads.js";
const databaseUrl = process.env.DATABASE_URL;
const describePostgres =
process.env.VERIFY_DATABASE && databaseUrl ? describe.sequential : describe.skip;
type Space = { userId: string; spaceId: string };
function actorOf(space: Space, isDeploymentOwner = false): Actor {
return {
userId: space.userId,
spaceId: space.spaceId,
email: `${space.userId}@rakazo.test`,
isDeploymentOwner,
};
}
function procedure(client: unknown, rpcPath: string) {
let node = client;
for (const key of rpcPath.split("/")) node = (node as Record<string, unknown>)[key];
return node as (input: unknown) => Promise<unknown>;
}
async function rpcCode(promise: Promise<unknown>): Promise<string> {
try {
await promise;
} catch (error) {
if (error instanceof ORPCError) return error.code;
throw error;
}
return "OK";
}
function sha256(bytes: Uint8Array): string {
return createHash("sha256").update(bytes).digest("hex");
}
describePostgres("media API authorization and uploads (PostgreSQL)", () => {
let db: MediaTestDb;
let repos: MediaRepos;
let mediaRoot: string;
let a: Space;
let b: Space;
let seededA: SeededMediaSpace;
let seededB: SeededMediaSpace;
let calls: MediaServiceCall[];
let renderOutputs: Map<string, string>;
let rpc: (actor: Actor) => unknown;
let app: Hono;
let currentActor: Actor | null;
function setup(options: { now?: () => Date; chunkTimeoutMs?: number } = {}) {
const fake = fakeMediaServices({
repos,
mediaRoot,
prisma: db.prisma,
profiles: [testProfile()],
renderOutputs,
...(options.now ? { now: options.now } : {}),
});
calls = fake.calls;
const router = createMediaRouter({
prisma: db.prisma,
services: fake.services,
dataDir: path.join(mediaRoot, ".."),
});
rpc = (actor) => createRouterClient(router, { context: { actor } });
app = new Hono();
mountMediaHttpRoutes(
app,
{
services: fake.services,
...(options.chunkTimeoutMs ? { chunkTimeoutMs: options.chunkTimeoutMs } : {}),
},
async () => currentActor,
);
}
async function snapshot(spaceId: string) {
const p = db.prisma;
const where = { spaceId };
const orderBy = { id: "asc" as const };
return JSON.stringify(
[
await p.mediaSource.findMany({ where, orderBy }),
await p.mediaAsset.findMany({ where, orderBy }),
await p.mediaUpload.findMany({ where, orderBy }),
await p.mediaSegment.findMany({ where, orderBy }),
await p.mediaAnalysis.findMany({ where, orderBy }),
await p.mediaCollection.findMany({ where, orderBy }),
await p.mediaCollectionItem.findMany({
where: { collection: where },
orderBy: [{ collectionId: "asc" }, { assetId: "asc" }],
}),
await p.mediaPerson.findMany({ where, orderBy }),
await p.mediaFaceTrack.findMany({ where, orderBy }),
await p.mediaMark.findMany({ where, orderBy }),
await p.mediaBrandKit.findMany({ where, orderBy }),
await p.mediaProject.findMany({ where, orderBy }),
await p.mediaEditPlan.findMany({ where, orderBy }),
await p.mediaRender.findMany({ where, orderBy }),
await p.mediaPublication.findMany({ where, orderBy }),
await p.mediaMetric.findMany({ where, orderBy }),
await p.mediaMetricMapping.findMany({ where, orderBy }),
],
(_key, value) => (typeof value === "bigint" ? value.toString() : value),
);
}
async function artifactIn(space: Space): Promise<string> {
const artifact = await db.prisma.artifact.create({
data: {
spaceId: space.spaceId,
userId: space.userId,
name: "consent.pdf",
mimeType: "application/pdf",
size: 1,
hash: "0".repeat(64),
storageKey: `test/${space.spaceId}/consent.pdf`,
},
});
return artifact.id;
}
/** A device item in `space` that waits for its original. */
async function remoteAsset(space: Space, index: number): Promise<string> {
const source = await procedure(
rpc(actorOf(space)),
"sources/registerClient",
)({ kind: "ios_library", label: "Phone", deviceId: `device-${index}` });
const synced = await repos.upsertSourceItems(space.spaceId, (source as { id: string }).id, [
sourceItem(index),
]);
const assetId = synced.changedAssetIds[0] as string;
await repos.requestUpload(space.spaceId, [assetId]);
return assetId;
}
function putChunk(uploadId: string, index: number, body: BodyInit, actor: Actor | null) {
currentActor = actor;
return app.request(`/api/media/uploads/${uploadId}/chunks/${index}`, { method: "PUT", body });
}
beforeAll(async () => {
db = openMediaTestDb(databaseUrl!);
repos = createMediaRepos(db.prisma);
const root = await mkdtemp(path.join(tmpdir(), "media-api-pg-"));
mediaRoot = path.join(root, "media");
await mkdir(mediaRoot);
renderOutputs = new Map();
a = await db.newUser();
b = await db.newUser();
seededA = await seedMediaSpace(repos, a.spaceId, a.userId);
seededB = await seedMediaSpace(repos, b.spaceId, b.userId);
setup();
});
afterAll(async () => {
await db?.cleanup();
if (mediaRoot) await rm(path.dirname(mediaRoot), { recursive: true, force: true });
});
it("refuses another space's ids in every media procedure and leaves that space untouched", async () => {
const before = await snapshot(a.spaceId);
const foreignIds = Object.values(seededA);
const intruder = rpc(actorOf(b));
const inputs = sampleMediaInputs(seededA, testBrandKit);
for (const [rpcPath, sample] of Object.entries(inputs)) {
if (sample.foreign === "dedicated") continue;
const call = procedure(intruder, rpcPath)(sample.input);
if (sample.foreign === "not_found") {
expect(await rpcCode(call), rpcPath).toBe("NOT_FOUND");
continue;
}
const result = JSON.stringify(await call);
for (const id of foreignIds) expect(result, rpcPath).not.toContain(id);
}
expect(await snapshot(a.spaceId)).toBe(before);
// No service ever received the other space's ids.
const leaked = calls.filter((call) =>
foreignIds.some((id) => JSON.stringify(call.input ?? null).includes(id)),
);
expect(leaked).toEqual([]);
expect(calls.every((call) => call.actor.spaceId === b.spaceId)).toBe(true);
});
it("serves the same inputs inside the caller's own space", async () => {
// Positive control: NOT_FOUND above comes from isolation, not from bad samples.
const fabricated = new Set(["collections/merge", "people/merge"]);
for (const [rpcPath, sample] of Object.entries(sampleMediaInputs(seededA, testBrandKit))) {
if (sample.foreign !== "not_found" || fabricated.has(rpcPath)) continue;
const space = await db.newUser();
const seeded = await seedMediaSpace(repos, space.spaceId, space.userId);
const input = sampleMediaInputs(seeded, testBrandKit)[rpcPath]?.input;
const code = await rpcCode(procedure(rpc(actorOf(space)), rpcPath)(input));
expect(code, rpcPath).not.toBe("NOT_FOUND");
}
}, 120_000);
it("refuses another space's records referenced from the caller's own ones", async () => {
const own = rpc(actorOf(b));
const foreignArtifact = await artifactIn(a);
const attempts: Array<[string, unknown]> = [
["projects/update", { projectId: seededB.projectId, collectionIds: [seededA.collectionId] }],
["projects/update", { projectId: seededB.projectId, brandKitId: seededA.brandKitId }],
[
"projects/create",
{
title: "Mixed",
goal: "teaser",
assetIds: [seededA.assetId],
profileIds: ["example-short-vertical"],
},
],
["people/assignTrack", { trackId: seededB.trackId, personId: seededA.personId }],
[
"marks/create",
{
kind: "tattoo",
assetId: seededB.assetId,
personId: seededA.personId,
prompt: { tMs: 0 },
treatment: "blur",
},
],
["marks/update", { markId: seededB.markId, personId: seededA.personId }],
[
"assets/setPaidOnly",
{ segmentIds: [seededB.segmentId, seededA.segmentId], paidOnly: true },
],
["collections/merge", { collectionIds: [seededB.collectionId, seededA.collectionId] }],
["people/merge", { personIds: [seededB.personId, seededA.personId] }],
["brandKits/upsert", { kit: { ...testBrandKit, logoArtifactIds: [foreignArtifact] } }],
[
"people/recordConsent",
{
personId: seededB.personId,
consentArtifactId: foreignArtifact,
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "document_on_file",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
},
],
];
const before = await snapshot(a.spaceId);
for (const [rpcPath, input] of attempts) {
expect(await rpcCode(procedure(own, rpcPath)(input)), rpcPath).toBe("NOT_FOUND");
}
expect(await snapshot(a.spaceId)).toBe(before);
const person = await db.prisma.mediaPerson.findUniqueOrThrow({
where: { id: seededB.personId },
});
expect(person.consentRecordedAt).toBeNull();
});
it("records consent only as the signed-in member and ignores it on updates", async () => {
const space = await db.newUser();
const seeded = await seedMediaSpace(repos, space.spaceId, space.userId);
const client = rpc(actorOf(space));
await procedure(
client,
"people/update",
)({
personId: seeded.personId,
displayName: "Person 1",
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
});
let person = await db.prisma.mediaPerson.findUniqueOrThrow({ where: { id: seeded.personId } });
expect(person.consentRecordedAt).toBeNull();
expect(person.ageVerifiedAt).toBeNull();
const artifactId = await artifactIn(space);
await procedure(
client,
"people/recordConsent",
)({
personId: seeded.personId,
consentArtifactId: artifactId,
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
});
person = await db.prisma.mediaPerson.findUniqueOrThrow({ where: { id: seeded.personId } });
expect(person.consentRecordedBy).toBe(space.userId);
expect(person.consentArtifactId).toBe(artifactId);
expect(person.ageVerificationMethod).toBe("platform_verified");
});
it("assembles a resumable chunked upload and hands it to ingest", async () => {
const assetId = await remoteAsset(b, 101);
const bytes = new TextEncoder().encode("0123456789");
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: bytes.length,
sha256: sha256(bytes),
fileName: "clip.mp4",
chunkBytes: 4,
});
const owner = actorOf(b);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), actorOf(a))).status).toBe(404);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), null)).status).toBe(401);
expect((await putChunk(upload.id, 3, bytes.slice(0, 2), owner)).status).toBe(400);
expect((await putChunk(upload.id, 0, bytes.slice(0, 5), owner)).status).toBe(413);
expect((await putChunk(upload.id, 0, bytes.slice(0, 3), owner)).status).toBe(400);
// Out of order and repeated chunks are fine; the client resumes from status.
expect((await putChunk(upload.id, 2, bytes.slice(8), owner)).status).toBe(200);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(200);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(200);
const client = rpc(owner);
const partial = (await procedure(client, "uploads/status")({ uploadId: upload.id })) as {
receivedChunks: number[];
};
expect([...partial.receivedChunks].sort()).toEqual([0, 2]);
expect(await rpcCode(procedure(client, "uploads/complete")({ uploadId: upload.id }))).toBe(
"CONFLICT",
);
expect((await putChunk(upload.id, 1, bytes.slice(4, 8), owner)).status).toBe(200);
calls.length = 0;
const done = (await procedure(client, "uploads/complete")({ uploadId: upload.id })) as {
status: string;
};
expect(done.status).toBe("complete");
const handed = calls.filter((call) => call.method === "uploadCompleted");
expect(handed).toHaveLength(1);
const input = handed[0]?.input as { tempPath: string; sha256: string; assetId: string };
expect(input.assetId).toBe(assetId);
expect(input.sha256).toBe(sha256(bytes));
expect(input.tempPath.startsWith(path.join(mediaRoot, b.spaceId, "uploads"))).toBe(true);
expect(await readFile(input.tempPath, "utf8")).toBe("0123456789");
const asset = await repos.getAssetRecord(b.spaceId, assetId);
expect(asset.storage).toBe("uploaded");
expect(asset.storagePath).toBe(`${b.spaceId}/uploads/${upload.id}/file`);
// Completing again is idempotent and does not hand the file over twice.
await procedure(client, "uploads/complete")({ uploadId: upload.id });
expect(calls.filter((call) => call.method === "uploadCompleted")).toHaveLength(1);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(409);
});
it("opens uploads through the RPC only for items that wait for their original", async () => {
const client = rpc(actorOf(b));
const assetId = await remoteAsset(b, 102);
const input = { assetId, sizeBytes: 10, sha256: "a".repeat(64), fileName: "clip.mp4" };
const first = (await procedure(client, "uploads/begin")(input)) as { id: string };
const resumed = (await procedure(client, "uploads/begin")(input)) as { id: string };
expect(resumed.id).toBe(first.id);
const referenced = { ...input, assetId: seededB.assetId };
expect(await rpcCode(procedure(client, "uploads/begin")(referenced))).toBe("CONFLICT");
const tooLarge = { ...input, sizeBytes: 21 * 1024 * 1024 * 1024 };
expect(await rpcCode(procedure(client, "uploads/begin")(tooLarge))).toBe("BAD_REQUEST");
});
it("fails an upload whose bytes do not match the announced checksum", async () => {
const assetId = await remoteAsset(b, 103);
const bytes = new TextEncoder().encode("abcdef");
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: bytes.length,
sha256: "f".repeat(64),
fileName: "clip.mp4",
chunkBytes: 4,
});
const owner = actorOf(b);
await putChunk(upload.id, 0, bytes.slice(0, 4), owner);
await putChunk(upload.id, 1, bytes.slice(4), owner);
const client = rpc(owner);
expect(await rpcCode(procedure(client, "uploads/complete")({ uploadId: upload.id }))).toBe(
"BAD_REQUEST",
);
const status = (await procedure(client, "uploads/status")({ uploadId: upload.id })) as {
status: string;
};
expect(status.status).toBe("failed");
const dir = path.join(mediaRoot, b.spaceId, "uploads", upload.id);
await expect(stat(dir)).rejects.toMatchObject({ code: "ENOENT" });
});
it("expires an idle upload and starts a fresh one", async () => {
const assetId = await remoteAsset(b, 104);
const input = { assetId, sizeBytes: 10, sha256: "b".repeat(64), fileName: "clip.mp4" };
const first = (await procedure(rpc(actorOf(b)), "uploads/begin")(input)) as { id: string };
setup({ now: () => new Date(Date.now() + MEDIA_UPLOAD_IDLE_MS + 60_000) });
try {
const response = await putChunk(first.id, 0, new Uint8Array(10), actorOf(b));
expect(response.status).toBe(409);
const status = (await procedure(
rpc(actorOf(b)),
"uploads/status",
)({
uploadId: first.id,
})) as { status: string };
expect(status.status).toBe("failed");
const fresh = (await procedure(rpc(actorOf(b)), "uploads/begin")(input)) as { id: string };
expect(fresh.id).not.toBe(first.id);
} finally {
setup();
}
});
it("drops a chunk that does not arrive in time", async () => {
const assetId = await remoteAsset(b, 105);
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: 10,
sha256: "c".repeat(64),
fileName: "clip.mp4",
});
setup({ chunkTimeoutMs: 50 });
try {
const stalled = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array(2));
},
});
currentActor = actorOf(b);
const response = await app.request(`/api/media/uploads/${upload.id}/chunks/0`, {
method: "PUT",
body: stalled,
duplex: "half",
} as RequestInit);
expect(response.status).toBe(408);
const status = await repos.uploadStatus(b.spaceId, upload.id);
expect(status.receivedChunks).toEqual([]);
} finally {
setup();
}
});
it("streams render files only inside the caller's space and render folder", async () => {
const renderDir = path.join(mediaRoot, a.spaceId, "renders", seededA.renderId);
await mkdir(renderDir, { recursive: true });
const video = path.join(renderDir, "video.mp4");
await writeFile(video, "0123456789");
renderOutputs.set(seededA.renderId, video);
const url = `/api/media/renders/${seededA.renderId}/files/video.mp4`;
currentActor = actorOf(a);
const full = await app.request(url);
expect(full.status).toBe(200);
expect(await full.text()).toBe("0123456789");
const ranged = await app.request(url, { headers: { range: "bytes=2-4" } });
expect(ranged.status).toBe(206);
expect(ranged.headers.get("content-range")).toBe("bytes 2-4/10");
expect(await ranged.text()).toBe("234");
const download = await app.request(`${url}?download=1`);
expect(download.headers.get("content-disposition")).toContain("attachment");
// A render that has not passed its gate never leaves as a download.
const queuedVideo = path.join(mediaRoot, a.spaceId, "renders", seededA.queuedRenderId, "v.mp4");
await mkdir(path.dirname(queuedVideo), { recursive: true });
await writeFile(queuedVideo, "x");
renderOutputs.set(seededA.queuedRenderId, queuedVideo);
const queuedUrl = `/api/media/renders/${seededA.queuedRenderId}/files/v.mp4`;
expect((await app.request(queuedUrl)).status).toBe(200);
expect((await app.request(`${queuedUrl}?download=1`)).status).toBe(404);
// A path the service reports outside the render folder is refused.
const outside = path.join(mediaRoot, a.spaceId, "video.mp4");
await writeFile(outside, "secret");
renderOutputs.set(seededA.renderId, outside);
expect((await app.request(url)).status).toBe(404);
renderOutputs.set(seededA.renderId, video);
currentActor = actorOf(b);
expect((await app.request(url)).status).toBe(404);
expect((await app.request(`/api/media/assets/${seededA.assetId}/thumb`)).status).toBe(404);
expect((await app.request(`/api/media/bundles/${seededA.publicationId}`)).status).toBe(404);
});
it("refuses a folder source to members who do not own the deployment", async () => {
const folder = await mkdtemp(path.join(tmpdir(), "media-footage-"));
try {
const input = { label: "Footage", path: folder };
const member = procedure(rpc(actorOf(b)), "sources/createFolder")(input);
expect(await rpcCode(member)).toBe("FORBIDDEN");
const created = (await procedure(rpc(actorOf(b, true)), "sources/createFolder")(input)) as {
id: string;
kind: string;
};
expect(created.kind).toBe("folder");
const row = await db.prisma.mediaSource.findUniqueOrThrow({ where: { id: created.id } });
expect(row.ownerUserId).toBe(b.userId);
} finally {
await rm(folder, { recursive: true, force: true });
}
});
});

333
apps/api/src/media.test.ts Normal file
View File

@ -0,0 +1,333 @@
import { mkdir, mkdtemp, rm, symlink } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createRouterClient, ORPCError } from "@orpc/server";
import type { Actor } from "@rakazo/contracts";
import { MediaToolInputSchemas, mediaRpcShapes } from "@rakazo/contracts";
import type { MediaRepos, PrismaClient } from "@rakazo/db";
import { InvalidMediaInputError, IsolationError, MediaStateConflictError } from "@rakazo/db";
import { PathOutsideRootError } from "@rakazo/media/node";
import { Hono } from "hono";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
createMediaRouter,
MEDIA_NOT_CONFIGURED,
mediaRpcError,
unconfiguredCapabilities,
validateMediaFolder,
} from "./media.js";
import { mountMediaHttpRoutes, parseByteRange } from "./media-http.js";
import type { MediaSampleIds } from "./media-test-fakes.js";
import { fakeMediaServices, mediaProcedurePaths, sampleMediaInputs } from "./media-test-fakes.js";
import { expectedChunkBytes, uploadChunkCount } from "./media-uploads.js";
const member: Actor = {
userId: "user-member",
spaceId: "space-a",
email: "member@rakazo.test",
isDeploymentOwner: false,
};
const owner: Actor = { ...member, userId: "user-owner", isDeploymentOwner: true };
const ids = Object.fromEntries(
[
"sourceId",
"assetId",
"secondAssetId",
"segmentId",
"trackId",
"personId",
"markId",
"collectionId",
"brandKitId",
"projectId",
"planId",
"renderId",
"publicationId",
"mappingId",
"uploadId",
].map((key) => [key, `id-${key.toLowerCase()}`]),
) as unknown as MediaSampleIds;
const kit = {
name: "Kit",
colors: {
primary: "#101010",
secondary: "#202020",
accent: "#303030",
text: "#f0f0f0",
background: "#000000",
},
};
function client(router: ReturnType<typeof createMediaRouter>, actor: Actor | null) {
return createRouterClient(router, { context: { actor } });
}
function procedure(target: unknown, rpcPath: string) {
let node: unknown = target;
for (const key of rpcPath.split("/")) node = (node as Record<string, unknown>)[key];
return node as (input: unknown) => Promise<unknown>;
}
async function rpcCode(promise: Promise<unknown>): Promise<string> {
try {
await promise;
} catch (error) {
if (error instanceof ORPCError) return error.code;
throw error;
}
return "OK";
}
/** Repositories that fail the test when reached. */
function untouchedRepos(): MediaRepos {
return new Proxy({} as MediaRepos, {
get: (_target, key) => () => {
throw new Error(`repository ${String(key)} must not be called`);
},
});
}
const prismaStub = {} as PrismaClient;
describe("media router without a configured sidecar", () => {
const router = createMediaRouter({ prisma: prismaStub, services: null, dataDir: "/data" });
const inputs = sampleMediaInputs(ids, kit);
it("has a sample input for every media procedure", () => {
expect(Object.keys(inputs).sort()).toEqual(mediaProcedurePaths().sort());
});
it("reports media as not configured so every surface keeps it hidden", async () => {
const capabilities = await client(router, member).capabilities({});
expect(capabilities).toEqual(unconfiguredCapabilities());
expect(capabilities.configured).toBe(false);
expect(
Object.values(capabilities.operations).every(
(op) => !(op as { available: boolean }).available,
),
).toBe(true);
});
it("refuses every other procedure", async () => {
for (const [rpcPath, sample] of Object.entries(inputs)) {
if (rpcPath === "capabilities") continue;
const promise = procedure(client(router, member), rpcPath)(sample.input);
await expect(promise, rpcPath).rejects.toMatchObject({
code: "PRECONDITION_FAILED",
message: MEDIA_NOT_CONFIGURED,
});
}
});
it("requires a signed-in member for every procedure", async () => {
for (const [rpcPath, sample] of Object.entries(inputs)) {
const code = await rpcCode(procedure(client(router, null), rpcPath)(sample.input));
expect(code, rpcPath).toBe("UNAUTHORIZED");
}
});
it("answers the byte routes with 404 and anonymous callers with 401", async () => {
const app = new Hono();
let actor: Actor | null = member;
mountMediaHttpRoutes(app, { services: null }, async () => actor);
const requests = [
new Request("http://api/api/media/uploads/u1/chunks/0", { method: "PUT", body: "abcd" }),
new Request("http://api/api/media/renders/r1/files/video.mp4"),
new Request("http://api/api/media/assets/a1/thumb"),
new Request("http://api/api/media/assets/a1/proxy"),
new Request("http://api/api/media/bundles/p1"),
];
for (const request of requests) {
expect((await app.request(request.clone())).status, request.url).toBe(404);
}
actor = null;
for (const request of requests) {
expect((await app.request(request.clone())).status, request.url).toBe(401);
}
});
});
describe("media router permissions", () => {
let root: string;
beforeAll(async () => {
root = await mkdtemp(path.join(tmpdir(), "media-api-"));
});
afterAll(async () => {
await rm(root, { recursive: true, force: true });
});
it("lets only the deployment owner add a folder source", async () => {
const created: unknown[] = [];
const repos = new Proxy({} as MediaRepos, {
get: (_target, key) => {
if (key === "createSource") {
return async (_spaceId: string, input: unknown) => {
created.push(input);
return {
id: "src-1",
kind: "folder",
label: "Footage",
status: "idle",
assetCount: 0,
lastSyncedAt: null,
error: null,
createdAt: "2026-10-01T00:00:00.000Z",
};
};
}
return () => {
throw new Error(`repository ${String(key)} must not be called`);
};
},
});
const { services, calls } = fakeMediaServices({ repos, mediaRoot: path.join(root, "media") });
const footage = path.join(root, "footage");
await mkdir(footage);
const router = createMediaRouter({
prisma: prismaStub,
services,
dataDir: path.join(root, "data"),
appRoot: path.join(root, "app"),
});
const denied = client(router, member).sources.createFolder({ label: "Footage", path: footage });
await expect(denied).rejects.toMatchObject({ code: "FORBIDDEN" });
expect(created).toEqual([]);
expect(calls).toEqual([]);
await client(router, owner).sources.createFolder({ label: "Footage", path: footage });
expect(created).toEqual([
{ kind: "folder", label: "Footage", ownerUserId: owner.userId, config: { path: footage } },
]);
expect(calls.map((call) => `${call.service}.${call.method}`)).toEqual(["ingest.requestSync"]);
});
it("accepts only a real, specific folder apart from evOo's own directories", async () => {
const dataDir = path.join(root, "data");
const appRoot = path.join(root, "app");
const folder = path.join(root, "clips");
await mkdir(path.join(dataDir, "media"), { recursive: true });
await mkdir(path.join(appRoot, "packages"), { recursive: true });
await mkdir(folder, { recursive: true });
await symlink(folder, path.join(root, "clips-link"));
const forbidden = { dataDir, appRoot };
await expect(validateMediaFolder(folder, forbidden)).resolves.toBe(folder);
const rejected = [
"clips",
path.join(root, "missing"),
path.join(root, "clips-link"),
"/",
dataDir,
path.join(dataDir, "media"),
root,
path.join(appRoot, "packages"),
];
for (const candidate of rejected) {
await expect(validateMediaFolder(candidate, forbidden), candidate).rejects.toBeInstanceOf(
InvalidMediaInputError,
);
}
});
});
describe("consent stays with the signed-in user", () => {
it("never accepts consent or age fields from the agent's people tool", () => {
const schema = MediaToolInputSchemas.media_people;
expect(schema.shape.action.options).toEqual(["list", "update", "merge"]);
const parsed = schema.parse({
action: "update",
personId: "person-1",
consentArtifactId: "artifact-1",
consentRecordedAt: "2026-09-01T00:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T00:00:00.000Z",
});
expect(Object.keys(parsed).sort()).toEqual(["action", "personId"]);
});
it("drops consent fields sent to people.update", async () => {
const patches: unknown[] = [];
const repos = new Proxy({} as MediaRepos, {
get: (_target, key) => {
if (key === "updatePerson") {
return async (_spaceId: string, _personId: string, patch: unknown) => {
patches.push(patch);
return {};
};
}
return () => {
throw new Error(`repository ${String(key)} must not be called`);
};
},
});
const { services } = fakeMediaServices({ repos, mediaRoot: "/unused" });
const router = createMediaRouter({ prisma: prismaStub, services, dataDir: "/data" });
await procedure(
client(router, member),
"people/update",
)({
personId: "person-1",
displayName: "Person 1",
consentRecordedAt: "2026-09-01T00:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T00:00:00.000Z",
}).catch(() => undefined);
expect(patches).toHaveLength(1);
expect(Object.entries(patches[0] as object).filter(([, value]) => value !== undefined)).toEqual(
[["displayName", "Person 1"]],
);
});
it("marks recordConsent as the only user-only write and keeps it out of the services", () => {
const people = mediaRpcShapes.people as Record<string, unknown>;
expect(Object.keys(people)).toContain("recordConsent");
const { services } = fakeMediaServices({ repos: untouchedRepos(), mediaRoot: "/unused" });
const serviceMethods = (["ingest", "privacy", "render", "marketing"] as const).flatMap((name) =>
Object.keys(services[name] as unknown as Record<string, unknown>),
);
expect(serviceMethods.filter((method) => /consent|age/i.test(method))).toEqual([]);
});
});
describe("media error mapping", () => {
it("never reveals whether a foreign id exists", () => {
expect((mediaRpcError(new IsolationError()) as ORPCError<string, unknown>).code).toBe(
"NOT_FOUND",
);
expect(
(mediaRpcError(new PathOutsideRootError("../x")) as ORPCError<string, unknown>).code,
).toBe("BAD_REQUEST");
expect(
(mediaRpcError(new MediaStateConflictError("busy")) as ORPCError<string, unknown>).code,
).toBe("CONFLICT");
const other = new Error("boom");
expect(mediaRpcError(other)).toBe(other);
});
});
describe("upload and streaming helpers", () => {
it("splits uploads into full chunks plus a shorter last one", () => {
const row = { sizeBytes: 10n, chunkBytes: 4 };
expect(uploadChunkCount(row)).toBe(3);
expect([0, 1, 2].map((index) => expectedChunkBytes(row, index))).toEqual([4, 4, 2]);
expect(uploadChunkCount({ sizeBytes: 8n, chunkBytes: 4 })).toBe(2);
expect(expectedChunkBytes({ sizeBytes: 8n, chunkBytes: 4 }, 1)).toBe(4);
});
it("parses single byte ranges and refuses unsatisfiable ones", () => {
expect(parseByteRange(undefined, 100)).toBeNull();
expect(parseByteRange("bytes=0-9", 100)).toEqual({ start: 0, end: 9 });
expect(parseByteRange("bytes=90-", 100)).toEqual({ start: 90, end: 99 });
expect(parseByteRange("bytes=-10", 100)).toEqual({ start: 90, end: 99 });
expect(parseByteRange("bytes=50-500", 100)).toEqual({ start: 50, end: 99 });
expect(parseByteRange("bytes=0-1,5-6", 100)).toBeNull();
expect(parseByteRange("bytes=100-", 100)).toBe("unsatisfiable");
expect(parseByteRange("bytes=9-3", 100)).toBe("unsatisfiable");
expect(parseByteRange("items=0-1", 100)).toBe("unsatisfiable");
expect(parseByteRange("bytes=-0", 100)).toBe("unsatisfiable");
});
});

552
apps/api/src/media.ts Normal file
View File

@ -0,0 +1,552 @@
import { lstat, realpath, rm } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { implement, ORPCError } from "@orpc/server";
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import type { MediaServices } from "@rakazo/adapters";
import { recordMetric, resetTeaserModel, saveBrandKit, segmentFunnel } from "@rakazo/adapters";
import type { Actor, BrandKit, MediaCapabilities, MediaSourceKind } from "@rakazo/contracts";
import { appContract, MediaInferenceOperationSchema } from "@rakazo/contracts";
import type { PrismaClient } from "@rakazo/db";
import {
InvalidMediaInputError,
InvalidProjectWorkspacePathError,
IsolationError,
lockMediaKey,
MediaStateConflictError,
mapMediaSource,
resolveProjectWorkspaceDirectory,
} from "@rakazo/db";
import { summarizeProfile } from "@rakazo/media";
import { PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import { beginMediaUpload, completeMediaUpload, UploadChunkError } from "./media-uploads.js";
export interface MediaRouterDeps {
prisma: PrismaClient;
/** Null when Media Studio is not configured: everything but capabilities refuses. */
services?: MediaServices | null;
/** DATA_DIR; a folder source may neither sit inside it nor contain it. */
dataDir: string;
/** Application install directory a folder source may not point into. */
appRoot?: string;
}
export const MEDIA_NOT_CONFIGURED = "Media Studio is not configured";
const DEFAULT_APP_ROOT = fileURLToPath(new URL("../../../", import.meta.url));
/** Translate domain errors into RPC errors; everything else passes through. */
export function mediaRpcError(error: unknown): unknown {
if (error instanceof ORPCError) return error;
if (error instanceof IsolationError) return new ORPCError("NOT_FOUND");
if (
error instanceof InvalidMediaInputError ||
error instanceof InvalidProjectWorkspacePathError
) {
return new ORPCError("BAD_REQUEST", { message: error.message });
}
if (error instanceof PathOutsideRootError) {
return new ORPCError("BAD_REQUEST", { message: "That path is not allowed." });
}
if (error instanceof MediaStateConflictError) {
return new ORPCError("CONFLICT", { message: error.message });
}
if (error instanceof UploadChunkError) {
return new ORPCError(error.status === 409 ? "CONFLICT" : "BAD_REQUEST", {
message: error.message,
});
}
if (error instanceof UnsupportedMediaOperationError) {
return new ORPCError("PRECONDITION_FAILED", { message: error.message });
}
return error;
}
export function unconfiguredCapabilities(): MediaCapabilities {
const unavailable = {
available: false,
experimental: false,
model: null,
license: null,
commercialUse: null,
};
return {
configured: false,
operations: Object.fromEntries(
MediaInferenceOperationSchema.options.map((operation) => [operation, unavailable]),
) as MediaCapabilities["operations"],
accelerator: "none",
hardwareEncoders: [],
};
}
function overlaps(a: string, b: string): boolean {
const inside = (root: string, candidate: string) => {
const relative = path.relative(root, candidate);
return relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
};
return inside(a, b) || inside(b, a);
}
async function realOrResolved(target: string): Promise<string> {
return realpath(target).catch(() => path.resolve(target));
}
/**
* A host folder for a folder source: absolute, an existing directory, not a
* symlink, and apart from DATA_DIR and the application itself. Returns the real path.
*/
export async function validateMediaFolder(
input: string,
forbidden: { dataDir: string; appRoot: string },
): Promise<string> {
const trimmed = input.trim();
if (!path.isAbsolute(trimmed)) {
throw new InvalidMediaInputError("Use an absolute folder path.");
}
const info = await lstat(trimmed).catch(() => null);
if (!info) throw new InvalidMediaInputError("That folder does not exist or is not readable.");
if (info.isSymbolicLink())
throw new InvalidMediaInputError("Choose the folder, not a link to it.");
const resolved = await resolveProjectWorkspaceDirectory(trimmed);
if (path.parse(resolved).root === resolved) {
throw new InvalidMediaInputError("Choose a specific folder.");
}
const dataDir = await realOrResolved(forbidden.dataDir);
const appRoot = await realOrResolved(forbidden.appRoot);
const relativeToApp = path.relative(appRoot, resolved);
const insideApp =
relativeToApp === "" || (!relativeToApp.startsWith("..") && !path.isAbsolute(relativeToApp));
if (overlaps(dataDir, resolved) || insideApp) {
throw new InvalidMediaInputError("That folder belongs to evOo itself.");
}
return resolved;
}
/** Artifacts referenced from media records must belong to the caller's space. */
export async function assertSpaceArtifacts(
prisma: PrismaClient,
spaceId: string,
artifactIds: string[],
): Promise<void> {
const ids = [...new Set(artifactIds)];
if (ids.length === 0) return;
const owned = await prisma.artifact.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
}
function brandKitArtifactIds(kit: BrandKit): string[] {
return [
...kit.fonts.map((font) => font.artifactId),
...kit.logoArtifactIds,
...(kit.lutArtifactId ? [kit.lutArtifactId] : []),
...kit.musicBeds.map((bed) => bed.artifactId),
];
}
async function removeMediaPath(mediaRoot: string, relative: string | null): Promise<void> {
if (!relative) return;
await rm(await resolveInside(mediaRoot, relative), { recursive: true, force: true });
}
export function createMediaRouter(deps: MediaRouterDeps) {
const os = implement(appContract.media).$context<{ actor: Actor | null; signal?: AbortSignal }>();
const authed = os.use(async ({ context, next }) => {
if (!context.actor) throw new ORPCError("UNAUTHORIZED");
try {
return await next({ context: { ...context, actor: context.actor } });
} catch (error) {
throw mediaRpcError(error);
}
});
const media = authed.use(async ({ next }) => {
if (!deps.services) {
throw new ORPCError("PRECONDITION_FAILED", { message: MEDIA_NOT_CONFIGURED });
}
return next({ context: { services: deps.services } });
});
const forbiddenRoots = { dataDir: deps.dataDir, appRoot: deps.appRoot ?? DEFAULT_APP_ROOT };
async function knownProfiles(services: MediaServices, profileIds: string[] | undefined) {
if (!profileIds) return;
const { profiles } = await services.deps.profiles();
const unknown = profileIds.filter((id) => !profiles.has(id));
if (unknown.length > 0) {
throw new InvalidMediaInputError(`Unknown profile: ${unknown.join(", ")}`);
}
}
return os.router({
capabilities: authed.capabilities.handler(async ({ context }) => {
const services = deps.services;
if (!services) return unconfiguredCapabilities();
const hardwareEncoders = await services.deps.renderer.hardwareEncoders().catch(() => []);
const inference = services.deps.inference;
if (!inference) return { ...unconfiguredCapabilities(), hardwareEncoders };
const capabilities = await inference
.capabilities(context.signal)
.catch(() => ({ ...unconfiguredCapabilities(), configured: true }));
return { ...capabilities, configured: true, hardwareEncoders };
}),
sources: {
list: media.sources.list.handler(({ context }) =>
context.services.deps.repos.listSources(context.actor.spaceId),
),
createFolder: media.sources.createFolder.handler(async ({ context, input }) => {
// A folder lives on the host, and the host is the deployment owner's machine.
if (!context.actor.isDeploymentOwner) throw new ORPCError("FORBIDDEN");
const folder = await validateMediaFolder(input.path, forbiddenRoots);
const { repos } = context.services.deps;
const source = await repos.createSource(context.actor.spaceId, {
kind: "folder",
label: input.label,
ownerUserId: context.actor.userId,
config: { path: folder },
});
await context.services.ingest.requestSync(context.actor, source.id);
return source;
}),
registerClient: media.sources.registerClient.handler(async ({ context, input }) => {
const { spaceId, userId } = context.actor;
const kind: MediaSourceKind = input.kind;
// One source per device: re-registering returns the existing one.
return deps.prisma.$transaction(async (tx) => {
await lockMediaKey(tx, `source:${spaceId}:${kind}:${input.deviceId}`);
const existing = await tx.mediaSource.findFirst({
where: { spaceId, kind, config: { path: ["deviceId"], equals: input.deviceId } },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
include: { _count: { select: { assets: true } } },
});
if (existing) return mapMediaSource(existing, existing._count.assets);
const created = await tx.mediaSource.create({
data: {
spaceId,
kind,
label: input.label,
ownerUserId: userId,
config: { deviceId: input.deviceId },
},
});
return mapMediaSource(created, 0);
});
}),
pushItems: media.sources.pushItems.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
return context.services.ingest.pushItems(context.actor, input);
}),
sync: media.sources.sync.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
await context.services.ingest.requestSync(context.actor, input.sourceId);
return { ok: true as const };
}),
remove: media.sources.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
await context.services.ingest.removeSource(context.actor, input.sourceId);
return { ok: true as const };
}),
},
assets: {
search: media.assets.search.handler(({ context, input }) =>
context.services.deps.repos.searchAssets(context.actor.spaceId, input),
),
get: media.assets.get.handler(({ context, input }) =>
context.services.deps.repos.getAsset(context.actor.spaceId, input.assetId),
),
setPaidOnly: media.assets.setPaidOnly.handler(async ({ context, input }) => {
await context.services.deps.repos.setSegmentsPaidOnly(
context.actor.spaceId,
input.segmentIds,
input.paidOnly,
);
return { ok: true as const };
}),
requestUpload: media.assets.requestUpload.handler(async ({ context, input }) => ({
awaiting: await context.services.deps.repos.requestUpload(
context.actor.spaceId,
input.assetIds,
),
})),
},
uploads: {
begin: media.uploads.begin.handler(({ context, input }) =>
beginMediaUpload(context.services, context.actor, input),
),
status: media.uploads.status.handler(({ context, input }) =>
context.services.deps.repos.uploadStatus(context.actor.spaceId, input.uploadId),
),
complete: media.uploads.complete.handler(({ context, input }) =>
completeMediaUpload(context.services, context.actor, input.uploadId),
),
},
collections: {
list: media.collections.list.handler(({ context, input }) =>
context.services.deps.repos.listCollections(context.actor.spaceId, input),
),
get: media.collections.get.handler(({ context, input }) =>
context.services.deps.repos.getCollection(context.actor.spaceId, input.collectionId),
),
rename: media.collections.rename.handler(({ context, input }) =>
context.services.deps.repos.renameCollection(
context.actor.spaceId,
input.collectionId,
input.title,
),
),
merge: media.collections.merge.handler(({ context, input }) =>
context.services.deps.repos.mergeCollections(context.actor.spaceId, input.collectionIds),
),
split: media.collections.split.handler(({ context, input }) =>
context.services.deps.repos.splitCollection(
context.actor.spaceId,
input.collectionId,
input.atAssetId,
),
),
createManual: media.collections.createManual.handler(({ context, input }) =>
context.services.deps.repos.createManualCollection(
context.actor.spaceId,
input.title,
input.assetIds,
),
),
recluster: media.collections.recluster.handler(async ({ context }) => {
await context.services.ingest.recluster(context.actor);
return { ok: true as const };
}),
},
people: {
list: media.people.list.handler(({ context }) =>
context.services.deps.repos.listPeople(context.actor.spaceId),
),
update: media.people.update.handler(({ context, input }) =>
context.services.deps.repos.updatePerson(context.actor.spaceId, input.personId, {
displayName: input.displayName,
isSelf: input.isSelf,
confirmed: input.confirmed,
anonymizeByDefault: input.anonymizeByDefault,
}),
),
merge: media.people.merge.handler(({ context, input }) =>
context.services.deps.repos.mergePeople(context.actor.spaceId, input.personIds),
),
assignTrack: media.people.assignTrack.handler(({ context, input }) =>
context.services.deps.repos.assignTrack(
context.actor.spaceId,
input.trackId,
input.personId,
),
),
// The only consent write path: the signed-in member is the recorder.
recordConsent: media.people.recordConsent.handler(async ({ context, input }) => {
await assertSpaceArtifacts(
deps.prisma,
context.actor.spaceId,
input.consentArtifactId ? [input.consentArtifactId] : [],
);
return context.services.deps.repos.recordConsent(
{ spaceId: context.actor.spaceId, recordedByUserId: context.actor.userId },
input,
);
}),
remove: media.people.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.getPerson(context.actor.spaceId, input.personId);
await context.services.privacy.removePerson(context.actor, input.personId);
return { ok: true as const };
}),
},
marks: {
list: media.marks.list.handler(({ context, input }) =>
context.services.deps.repos.listMarks(context.actor.spaceId, {
...(input.personId ? { personId: input.personId } : {}),
...(input.assetId ? { assetId: input.assetId } : {}),
}),
),
create: media.marks.create.handler(async ({ context, input }) => {
const { repos } = context.services.deps;
await repos.getAssetRecord(context.actor.spaceId, input.assetId);
if (input.personId) await repos.getPerson(context.actor.spaceId, input.personId);
return context.services.privacy.createMark(context.actor, input);
}),
update: media.marks.update.handler(async ({ context, input }) => {
const { repos } = context.services.deps;
await repos.getMark(context.actor.spaceId, input.markId);
if (input.personId) await repos.getPerson(context.actor.spaceId, input.personId);
return context.services.privacy.updateMark(context.actor, input);
}),
remove: media.marks.remove.handler(async ({ context, input }) => {
const { repos, mediaRoot } = context.services.deps;
const { maskDir } = await repos.removeMark(context.actor.spaceId, input.markId);
await removeMediaPath(mediaRoot, maskDir);
return { ok: true as const };
}),
},
brandKits: {
list: media.brandKits.list.handler(({ context }) =>
context.services.deps.repos.listBrandKits(context.actor.spaceId),
),
upsert: media.brandKits.upsert.handler(async ({ context, input }) => {
await assertSpaceArtifacts(
deps.prisma,
context.actor.spaceId,
brandKitArtifactIds(input.kit),
);
return saveBrandKit(context.services.deps, context.actor, input);
}),
remove: media.brandKits.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.removeBrandKit(context.actor.spaceId, input.id);
return { ok: true as const };
}),
templates: media.brandKits.templates.handler(({ context }) =>
context.services.deps.motion.templates(),
),
introVariants: media.brandKits.introVariants.handler(async ({ context, input }) => {
await context.services.deps.repos.getBrandKit(context.actor.spaceId, input.id);
return context.services.marketing.introVariants(context.actor, {
brandKitId: input.id,
count: input.count,
});
}),
},
profiles: {
list: media.profiles.list.handler(async ({ context }) => {
const { profiles } = await context.services.deps.profiles();
const now = context.services.deps.now();
return [...profiles.values()].map(({ profile, origin }) =>
summarizeProfile(profile, origin, now),
);
}),
get: media.profiles.get.handler(async ({ context, input }) => {
const { profiles } = await context.services.deps.profiles();
const entry = profiles.get(input.profileId);
if (!entry) throw new ORPCError("NOT_FOUND");
return entry.profile;
}),
},
projects: {
list: media.projects.list.handler(({ context, input }) =>
context.services.deps.repos.listProjects(context.actor.spaceId, input),
),
get: media.projects.get.handler(({ context, input }) =>
context.services.deps.repos.getProject(context.actor.spaceId, input.projectId),
),
create: media.projects.create.handler(async ({ context, input }) => {
await knownProfiles(context.services, input.profileIds);
return context.services.deps.repos.createProject(
context.actor.spaceId,
context.actor.userId,
input,
);
}),
update: media.projects.update.handler(async ({ context, input }) => {
await knownProfiles(context.services, input.profileIds);
const { projectId, ...patch } = input;
return context.services.deps.repos.updateProject(context.actor.spaceId, projectId, patch);
}),
remove: media.projects.remove.handler(async ({ context, input }) => {
const { repos, mediaRoot } = context.services.deps;
const { spaceId } = context.actor;
await repos.getProjectRecord(spaceId, input.projectId);
const renders = await repos.listRenders(spaceId, input.projectId);
await repos.removeProject(spaceId, input.projectId);
for (const render of renders)
await removeMediaPath(mediaRoot, `${spaceId}/renders/${render.id}`);
return { ok: true as const };
}),
autoPlan: media.projects.autoPlan.handler(async ({ context, input }) => {
await context.services.deps.repos.getProjectRecord(context.actor.spaceId, input.projectId);
await knownProfiles(context.services, input.profileIds);
return context.services.marketing.autoPlan(context.actor, input);
}),
},
plans: {
submit: media.plans.submit.handler(async ({ context, input }) => {
await context.services.deps.repos.getProjectRecord(context.actor.spaceId, input.projectId);
return context.services.marketing.submitPlan(context.actor, {
projectId: input.projectId,
edl: input.edl,
rationale: input.rationale,
createdBy: "user",
});
}),
},
renders: {
start: media.renders.start.handler(async ({ context, input }) => {
await context.services.deps.repos.getPlan(context.actor.spaceId, input.planId);
return context.services.render.start(context.actor, input);
}),
get: media.renders.get.handler(({ context, input }) =>
context.services.deps.repos.getRender(context.actor.spaceId, input.renderId),
),
cancel: media.renders.cancel.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.render.cancel(context.actor, input.renderId);
}),
thumbnails: media.renders.thumbnails.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.thumbnails(context.actor, input);
}),
},
marketing: {
copy: media.marketing.copy.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.copy(context.actor, input);
}),
},
publications: {
list: media.publications.list.handler(({ context, input }) =>
context.services.deps.repos.listPublications(context.actor.spaceId, input),
),
create: media.publications.create.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.createPublication(context.actor, input);
}),
bundle: media.publications.bundle.handler(async ({ context, input }) => {
await context.services.deps.repos.getPublication(
context.actor.spaceId,
input.publicationId,
);
return context.services.marketing.bundle(context.actor, input.publicationId);
}),
markPublished: media.publications.markPublished.handler(({ context, input }) =>
context.services.deps.repos.markPublicationPublished(
context.actor.spaceId,
input.publicationId,
input.externalUrl,
context.services.deps.now(),
),
),
cancel: media.publications.cancel.handler(async ({ context, input }) => {
await context.services.deps.repos.getPublication(
context.actor.spaceId,
input.publicationId,
);
return context.services.marketing.cancelPublication(context.actor, input.publicationId);
}),
},
metrics: {
mappings: media.metrics.mappings.handler(({ context }) =>
context.services.deps.repos.listMetricMappings(context.actor.spaceId),
),
saveMapping: media.metrics.saveMapping.handler(({ context, input }) =>
context.services.deps.repos.saveMetricMapping(context.actor.spaceId, input),
),
importCsv: media.metrics.importCsv.handler(async ({ context, input }) => {
await context.services.deps.repos.getMetricMapping(context.actor.spaceId, input.mappingId);
return context.services.marketing.importMetricsCsv(context.actor, input);
}),
record: media.metrics.record.handler(({ context, input }) =>
recordMetric(context.services.deps, context.actor, input),
),
funnel: media.metrics.funnel.handler(({ context, input }) =>
context.services.deps.repos.funnel(context.actor.spaceId, input.projectId),
),
segments: media.metrics.segments.handler(({ context, input }) =>
segmentFunnel(context.services.deps, context.actor, input.projectId),
),
resetTeaserModel: media.metrics.resetTeaserModel.handler(async ({ context }) => {
await resetTeaserModel(context.services.deps, context.actor);
return { ok: true as const };
}),
},
});
}

View File

@ -24,6 +24,7 @@ import type {
ConnectorRegistry,
EncryptedSecretStore,
IntegrationProviderSettings,
MediaServices,
MemoryProviderResolver,
PiOAuthLogins,
RemoteConnectorDependencies,
@ -166,6 +167,7 @@ import {
renameMainConversation,
} from "./main-conversation.js";
import { buildMcpUpdateMaterial } from "./mcp-material.js";
import { createMediaRouter } from "./media.js";
import {
disconnectMemoryProvider,
getMemoryProviderConfig,
@ -463,6 +465,8 @@ export interface RouterDeps {
remoteConnectors?: RemoteConnectorDependencies;
artifacts: ArtifactStore;
dataDir: string;
/** Null when no media sidecar (or scripted emulator) is configured; Media Studio then stays hidden. */
mediaServices?: MediaServices | null;
/** Present when the external messaging surface is enabled. */
messaging?: { enabled: boolean; providers: string[]; openSignup: boolean };
env: {
@ -526,6 +530,11 @@ export function createRouter(deps: RouterDeps) {
});
return os.router({
media: createMediaRouter({
prisma: deps.prisma,
services: deps.mediaServices ?? null,
dataDir: deps.dataDir,
}),
aiConsent: {
status: authed.aiConsent.status.handler(({ context, input }) =>
aiConsentStatus(deps, context.actor, input),

View File

@ -0,0 +1,2 @@
.build/
.swiftpm/

View File

@ -0,0 +1,18 @@
// swift-tools-version:5.9
import PackageDescription
// PhotosHelperCore holds everything that does not touch PhotoKit (argument parsing,
// cursors, JSON envelopes, the export path guard, hashing) so XCTest runs without
// Photos access. The executable target is the thin PhotoKit layer.
let package = Package(
name: "photos-helper",
platforms: [.macOS(.v13)],
products: [
.executable(name: "photos-helper", targets: ["photos-helper"])
],
targets: [
.target(name: "PhotosHelperCore"),
.executableTarget(name: "photos-helper", dependencies: ["PhotosHelperCore"]),
.testTarget(name: "PhotosHelperCoreTests", dependencies: ["PhotosHelperCore"]),
]
)

View File

@ -0,0 +1,142 @@
import Foundation
public enum ExportVariant: String, Equatable {
/// The untouched original resource, downloaded from iCloud when needed.
case original
/// A smaller rendition of the current version (720p video, 1280 px JPEG).
case proxy
}
public enum Command: Equatable {
/// `prompt: false` only reads the current status and never shows the system dialog.
case authorize(prompt: Bool)
case list(cursor: ListCursor?, limit: Int, since: Date?)
case asset(localIdentifier: String)
case export(localIdentifier: String, destination: String, variant: ExportVariant)
case changes(token: String)
}
public let usageText = """
usage: photos-helper authorize [--no-prompt]
photos-helper list [--cursor <opaque>] [--limit <1-1000>] [--since <iso8601>]
photos-helper asset <localIdentifier>
photos-helper export <localIdentifier> --dest <dir> [--original|--proxy]
photos-helper changes --token <opaque>
"""
/// Parses argv without the program name. Unknown flags, repeated flags and missing
/// values are usage errors so a typo never silently changes behaviour.
public func parseArguments(_ arguments: [String]) throws -> Command {
guard let name = arguments.first else { throw HelperError(.usage, usageText) }
var options = try OptionReader(Array(arguments.dropFirst()))
switch name {
case "authorize":
let noPrompt = options.flag("--no-prompt")
try options.finish(positionals: 0)
return .authorize(prompt: !noPrompt)
case "list":
let cursor = try options.value("--cursor").map(ListCursor.decode)
let limit = try options.value("--limit").map(parseLimit) ?? Limits.defaultPage
let since = try options.value("--since").map { value -> Date in
guard let date = parseTimestamp(value) else {
throw HelperError(.usage, "--since must be an ISO 8601 timestamp.")
}
return date
}
try options.finish(positionals: 0)
return .list(cursor: cursor, limit: limit, since: since)
case "asset":
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .asset(localIdentifier: identifier)
case "export":
guard let destination = options.value("--dest") else {
throw HelperError(.usage, "export needs --dest <dir>.")
}
let original = options.flag("--original")
let proxy = options.flag("--proxy")
if original && proxy { throw HelperError(.usage, "Choose --original or --proxy, not both.") }
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .export(
localIdentifier: identifier,
destination: destination,
variant: proxy ? .proxy : .original)
case "changes":
guard let token = options.value("--token"), !token.isEmpty else {
throw HelperError(.usage, "changes needs --token <opaque>.")
}
try options.finish(positionals: 0)
return .changes(token: token)
default:
throw HelperError(.usage, usageText)
}
}
func parseLimit(_ value: String) throws -> Int {
guard let limit = Int(value), (1...Limits.maxPage).contains(limit) else {
throw HelperError(.usage, "--limit must be between 1 and \(Limits.maxPage).")
}
return limit
}
func validIdentifier(_ value: String) throws -> String {
guard !value.isEmpty, value.utf16.count <= Limits.externalId, !value.hasPrefix("-"),
!value.unicodeScalars.contains(where: { CharacterSet.controlCharacters.contains($0) })
else { throw HelperError(.usage, "Invalid local identifier.") }
return value
}
private struct OptionReader {
private var values: [String: String] = [:]
private var flags: Set<String> = []
private var consumed: Set<String> = []
private(set) var positionals: [String] = []
private static let valueOptions: Set<String> = ["--cursor", "--limit", "--since", "--dest", "--token"]
private static let flagOptions: Set<String> = ["--no-prompt", "--original", "--proxy"]
init(_ arguments: [String]) throws {
var index = 0
while index < arguments.count {
let argument = arguments[index]
if Self.valueOptions.contains(argument) {
guard index + 1 < arguments.count else {
throw HelperError(.usage, "\(argument) needs a value.")
}
guard values[argument] == nil else { throw HelperError(.usage, "\(argument) given twice.") }
values[argument] = arguments[index + 1]
index += 2
} else if Self.flagOptions.contains(argument) {
guard !flags.contains(argument) else { throw HelperError(.usage, "\(argument) given twice.") }
flags.insert(argument)
index += 1
} else if argument.hasPrefix("--") {
throw HelperError(.usage, "Unknown option \(argument).")
} else {
positionals.append(argument)
index += 1
}
}
}
mutating func value(_ name: String) -> String? {
consumed.insert(name)
return values[name]
}
mutating func flag(_ name: String) -> Bool {
consumed.insert(name)
return flags.contains(name)
}
/// Rejects options that belong to another command and surplus positionals.
func finish(positionals expected: Int) throws {
for name in Set(values.keys).union(flags) where !consumed.contains(name) {
throw HelperError(.usage, "\(name) is not valid here.")
}
guard positionals.count == expected else {
throw HelperError(.usage, expected == 0 ? "Unexpected argument." : "Expected \(expected) argument.")
}
}
}

View File

@ -0,0 +1,37 @@
import Foundation
/// A failure the desktop app maps by `code`. Messages never contain library content
/// beyond the identifier the caller passed in.
public struct HelperError: Error, Equatable {
public enum Code: String, Codable, CaseIterable {
case usage
case notAuthorized = "not_authorized"
case notFound = "not_found"
case pathRejected = "path_rejected"
case destinationExists = "destination_exists"
case network
case exportFailed = "export_failed"
case `internal`
}
public let code: Code
public let message: String
public init(_ code: Code, _ message: String) {
self.code = code
self.message = message
}
/// sysexits(3) style codes so a shell user can tell failure classes apart.
public var exitCode: Int32 {
switch code {
case .usage: return 64
case .notFound: return 66
case .pathRejected, .destinationExists: return 73
case .exportFailed: return 74
case .network: return 75
case .notAuthorized: return 77
case .internal: return 70
}
}
}

View File

@ -0,0 +1,249 @@
import Foundation
/// Mirrors `ClientMediaItemSchema` in packages/contracts/src/media.ts. Every key is
/// always present (nullable fields encode as `null`) because the schema has no optional keys.
public struct ClientMediaItem: Encodable, Equatable {
public enum Kind: String, Encodable {
case video
case photo
case livePhoto = "live_photo"
}
public var externalId: String
public var kind: Kind
public var capturedAt: String?
public var durationMs: Int?
public var width: Int?
public var height: Int?
public var fps: Double?
public var gpsLat: Double?
public var gpsLon: Double?
public var deviceModel: String?
public var albumNames: [String]
public var isFavorite: Bool
public var fileName: String?
public init(
externalId: String,
kind: Kind,
capturedAt: Date?,
durationSeconds: Double?,
width: Int?,
height: Int?,
latitude: Double?,
longitude: Double?,
albumNames: [String],
isFavorite: Bool,
fileName: String?
) {
self.externalId = externalId
self.kind = kind
self.capturedAt = capturedAt.map(formatTimestamp)
if let durationSeconds, durationSeconds.isFinite, durationSeconds > 0 {
durationMs = Int((durationSeconds * 1000).rounded())
} else {
durationMs = nil
}
self.width = width.flatMap { $0 > 0 ? $0 : nil }
self.height = height.flatMap { $0 > 0 ? $0 : nil }
// PhotoKit exposes neither frame rate nor camera model without loading the
// file; the server reads both with ffprobe after upload.
fps = nil
deviceModel = nil
let coordinate = roundedCoordinate(latitude: latitude, longitude: longitude)
gpsLat = coordinate?.latitude
gpsLon = coordinate?.longitude
self.albumNames = Array(
albumNames.map { clip($0, maxCharacters: Limits.albumName) }
.filter { !$0.isEmpty }
.prefix(Limits.albumCount))
self.isFavorite = isFavorite
self.fileName = fileName.map { clip($0, maxCharacters: Limits.fileName) }
}
enum CodingKeys: String, CodingKey {
case externalId, kind, capturedAt, durationMs, width, height, fps, gpsLat, gpsLon
case deviceModel, albumNames, isFavorite, fileName
}
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(externalId, forKey: .externalId)
try container.encode(kind, forKey: .kind)
// `encode` on an Optional writes `null`, unlike the synthesized `encodeIfPresent`.
try container.encode(capturedAt, forKey: .capturedAt)
try container.encode(durationMs, forKey: .durationMs)
try container.encode(width, forKey: .width)
try container.encode(height, forKey: .height)
try container.encode(fps, forKey: .fps)
try container.encode(gpsLat, forKey: .gpsLat)
try container.encode(gpsLon, forKey: .gpsLon)
try container.encode(deviceModel, forKey: .deviceModel)
try container.encode(albumNames, forKey: .albumNames)
try container.encode(isFavorite, forKey: .isFavorite)
try container.encode(fileName, forKey: .fileName)
}
}
/// Bounds from the contract schema, applied here so one odd asset cannot make a whole
/// page fail validation in the app.
public enum Limits {
public static let externalId = 200
public static let albumName = 200
public static let albumCount = 50
public static let fileName = 255
public static let defaultPage = 500
public static let maxPage = 1000
/// Above this many changed assets a full relisting is cheaper than one huge document.
public static let maxChanges = 5000
}
/// Clips to `maxCharacters` UTF-16 units (what zod's `.max()` counts) without splitting
/// a character.
public func clip(_ value: String, maxCharacters: Int) -> String {
if value.utf16.count <= maxCharacters { return value }
var result = ""
var units = 0
for character in value {
units += character.utf16.count
if units > maxCharacters { break }
result.append(character)
}
return result
}
/// Two decimals is about 1 km, matching `roundCoordinate` in @rakazo/media. Precise
/// locations never leave the device.
public func roundedCoordinate(latitude: Double?, longitude: Double?) -> (
latitude: Double, longitude: Double
)? {
guard let latitude, let longitude, latitude.isFinite, longitude.isFinite,
(-90...90).contains(latitude), (-180...180).contains(longitude),
!(latitude == 0 && longitude == 0)
else { return nil }
return ((latitude * 100).rounded() / 100, (longitude * 100).rounded() / 100)
}
/// RFC 3339 in UTC with milliseconds, accepted by zod's `datetime({ offset: true })`.
public func formatTimestamp(_ date: Date) -> String {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
formatter.timeZone = TimeZone(identifier: "UTC")
return formatter.string(from: date)
}
public func parseTimestamp(_ value: String) -> Date? {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
if let date = formatter.date(from: value) { return date }
formatter.formatOptions = [.withInternetDateTime]
return formatter.date(from: value)
}
public enum AuthorizationState: String, Encodable {
case authorized, limited, denied, restricted, notDetermined
}
public struct AuthorizationOutput: Encodable, Equatable {
public let status: AuthorizationState
public init(status: AuthorizationState) { self.status = status }
}
public struct ListOutput: Encodable {
public let items: [ClientMediaItem]
public let nextCursor: String?
/// Library change token captured before enumeration; `changes --token` resumes from it.
public let changeToken: String?
public init(items: [ClientMediaItem], nextCursor: String?, changeToken: String?) {
self.items = items
self.nextCursor = nextCursor
self.changeToken = changeToken
}
enum CodingKeys: String, CodingKey { case items, nextCursor, changeToken }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(items, forKey: .items)
try container.encode(nextCursor, forKey: .nextCursor)
try container.encode(changeToken, forKey: .changeToken)
}
}
public struct AssetResourceInfo: Encodable, Equatable {
public let type: String
public let fileName: String
public let uti: String
public init(type: String, fileName: String, uti: String) {
self.type = type
self.fileName = fileName
self.uti = uti
}
}
public struct AssetOutput: Encodable {
public let item: ClientMediaItem
public let modifiedAt: String?
public let hidden: Bool
public let resources: [AssetResourceInfo]
public init(item: ClientMediaItem, modifiedAt: Date?, hidden: Bool, resources: [AssetResourceInfo]) {
self.item = item
self.modifiedAt = modifiedAt.map(formatTimestamp)
self.hidden = hidden
self.resources = resources
}
enum CodingKeys: String, CodingKey { case item, modifiedAt, hidden, resources }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(item, forKey: .item)
try container.encode(modifiedAt, forKey: .modifiedAt)
try container.encode(hidden, forKey: .hidden)
try container.encode(resources, forKey: .resources)
}
}
public struct ChangesOutput: Encodable {
/// Token to store for the next call; null only when a full relisting is required.
public let token: String?
public let changed: [ClientMediaItem]
public let deleted: [String]
public let fullResyncRequired: Bool
public init(token: String?, changed: [ClientMediaItem], deleted: [String], fullResyncRequired: Bool) {
self.token = token
self.changed = changed
self.deleted = deleted
self.fullResyncRequired = fullResyncRequired
}
public static func fullResync() -> ChangesOutput {
ChangesOutput(token: nil, changed: [], deleted: [], fullResyncRequired: true)
}
enum CodingKeys: String, CodingKey { case token, changed, deleted, fullResyncRequired }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(token, forKey: .token)
try container.encode(changed, forKey: .changed)
try container.encode(deleted, forKey: .deleted)
try container.encode(fullResyncRequired, forKey: .fullResyncRequired)
}
}
public struct ExportOutput: Encodable, Equatable {
public let path: String
public let fileName: String
public let sizeBytes: Int64
public let sha256: String
public init(path: String, fileName: String, sizeBytes: Int64, sha256: String) {
self.path = path
self.fileName = fileName
self.sizeBytes = sizeBytes
self.sha256 = sha256
}
}

View File

@ -0,0 +1,56 @@
import CryptoKit
import Foundation
/// stdout carries exactly one JSON document per successful command; stderr carries
/// JSON lines (`{"progress":…}` while exporting, `{"error":…}` on failure).
public func encodeJSON<T: Encodable>(_ value: T) throws -> String {
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes]
encoder.nonConformingFloatEncodingStrategy = .throw
let data = try encoder.encode(value)
guard let text = String(data: data, encoding: .utf8) else {
throw HelperError(.internal, "Output was not UTF-8.")
}
return text
}
private struct ErrorEnvelope: Encodable {
struct Body: Encodable {
let code: String
let message: String
}
let error: Body
}
public func errorLine(_ error: HelperError) -> String {
let envelope = ErrorEnvelope(error: .init(code: error.code.rawValue, message: error.message))
return (try? encodeJSON(envelope))
?? #"{"error":{"code":"internal","message":"Could not encode the error."}}"#
}
private struct ProgressLine: Encodable {
let progress: Double
}
/// Progress clamped to 0…1 and rounded to three decimals so the line stays short.
public func progressLine(_ fraction: Double) -> String {
let value = fraction.isFinite ? min(1, max(0, fraction)) : 0
return (try? encodeJSON(ProgressLine(progress: (value * 1000).rounded() / 1000)))
?? #"{"progress":0}"#
}
/// Streams the file through SHA-256 in 1 MiB reads so large videos never load fully.
public func sha256Hex(of url: URL) throws -> (hex: String, sizeBytes: Int64) {
let handle = try FileHandle(forReadingFrom: url)
defer { try? handle.close() }
var hasher = SHA256()
var size: Int64 = 0
while true {
let chunk = try handle.read(upToCount: 1 << 20) ?? Data()
if chunk.isEmpty { break }
hasher.update(data: chunk)
size += Int64(chunk.count)
}
let hex = hasher.finalize().map { String(format: "%02x", $0) }.joined()
return (hex, size)
}

View File

@ -0,0 +1,131 @@
import Foundation
/// Listing order: undated assets first, then creation date ascending, ties broken by
/// local identifier. PhotoKit can only sort by date, so ties are ordered here.
public struct SortKey: Comparable, Hashable {
/// Milliseconds since 1970; nil for assets without a creation date.
public let creationMillis: Int64?
public let localIdentifier: String
public init(creationMillis: Int64?, localIdentifier: String) {
self.creationMillis = creationMillis
self.localIdentifier = localIdentifier
}
public init(creationDate: Date?, localIdentifier: String) {
self.init(
creationMillis: creationDate.map { Int64(($0.timeIntervalSince1970 * 1000).rounded(.down)) },
localIdentifier: localIdentifier)
}
public static func < (lhs: SortKey, rhs: SortKey) -> Bool {
if lhs.creationMillis != rhs.creationMillis {
switch (lhs.creationMillis, rhs.creationMillis) {
case (nil, _): return true
case (_, nil): return false
case let (left?, right?): return left < right
}
}
return lhs.localIdentifier < rhs.localIdentifier
}
}
/// Opaque to the app: base64url of `{"v":1,"d":<millis|null>,"id":"<localIdentifier>"}`.
/// The cursor names the last item returned, so listing resumes after it even when
/// assets were added or removed in between.
public struct ListCursor: Equatable {
public let key: SortKey
public init(key: SortKey) { self.key = key }
private struct Payload: Codable {
let v: Int
let d: Int64?
let id: String
enum CodingKeys: String, CodingKey { case v, d, id }
func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(v, forKey: .v)
try container.encode(d, forKey: .d)
try container.encode(id, forKey: .id)
}
}
public func encoded() -> String {
let payload = Payload(v: 1, d: key.creationMillis, id: key.localIdentifier)
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys]
// Encoding a struct of plain values cannot fail.
let data = (try? encoder.encode(payload)) ?? Data()
return base64URLEncode(data)
}
public static func decode(_ value: String) throws -> ListCursor {
guard value.count <= 2048, let data = base64URLDecode(value),
let payload = try? JSONDecoder().decode(Payload.self, from: data),
payload.v == 1, !payload.id.isEmpty
else { throw HelperError(.usage, "Invalid cursor.") }
return ListCursor(key: SortKey(creationMillis: payload.d, localIdentifier: payload.id))
}
}
public func base64URLEncode(_ data: Data) -> String {
data.base64EncodedString()
.replacingOccurrences(of: "+", with: "-")
.replacingOccurrences(of: "/", with: "_")
.replacingOccurrences(of: "=", with: "")
}
public func base64URLDecode(_ value: String) -> Data? {
guard value.allSatisfy({ $0.isASCII && ($0.isLetter || $0.isNumber || $0 == "-" || $0 == "_") })
else { return nil }
var base64 = value.replacingOccurrences(of: "-", with: "+").replacingOccurrences(of: "_", with: "/")
while base64.count % 4 != 0 { base64.append("=") }
return Data(base64Encoded: base64)
}
public struct Page {
/// Positions in the source sequence, in listing order.
public let indices: [Int]
public let hasMore: Bool
}
/// Selects the next page from a sequence sorted by creation date only (ties in any
/// order). Binary search finds the first candidate date; a whole same-date group is
/// collected before sorting so tie order never depends on PhotoKit.
public func nextPage(count: Int, key: (Int) -> SortKey, after cursor: ListCursor?, limit: Int)
-> Page
{
precondition(limit > 0)
var start = 0
if let cursor, let cursorMillis = cursor.key.creationMillis {
var low = 0
var high = count
while low < high {
let mid = (low + high) / 2
let millis = key(mid).creationMillis
if millis == nil || millis! < cursorMillis { low = mid + 1 } else { high = mid }
}
start = low
}
var candidates: [(index: Int, key: SortKey)] = []
var stoppedEarly = false
var index = start
while index < count {
let current = key(index)
if candidates.count >= limit, let last = candidates.last, current.creationMillis != last.key.creationMillis {
// Dates ascend, so everything from here on sorts after the collected group.
stoppedEarly = true
break
}
if cursor == nil || cursor!.key < current {
candidates.append((index, current))
}
index += 1
}
candidates.sort { $0.key < $1.key }
let hasMore = stoppedEarly || candidates.count > limit
return Page(indices: candidates.prefix(limit).map(\.index), hasMore: hasMore)
}

View File

@ -0,0 +1,72 @@
import Foundation
/// Where an export is written: a final path and a sibling temporary path inside the
/// destination directory.
public struct ExportDestination: Equatable {
public let directory: URL
public let finalURL: URL
public let temporaryURL: URL
public let fileName: String
}
/// Resolves the export target. The directory must be absolute, free of `..`, existing
/// and a directory after resolving symlinks (macOS temp dirs live behind /var). The file
/// name comes from Photos, so it is reduced to one safe component and the joined path is
/// checked to stay a direct child of the resolved directory. Existing files are never
/// overwritten.
public func resolveExportDestination(
directory: String,
fileName rawFileName: String,
temporarySuffix: String = UUID().uuidString,
fileManager: FileManager = .default
) throws -> ExportDestination {
guard directory.hasPrefix("/"), !directory.contains("\0") else {
throw HelperError(.pathRejected, "--dest must be an absolute directory path.")
}
let components = directory.split(separator: "/", omittingEmptySubsequences: true)
guard !components.contains(where: { $0 == ".." || $0 == "." }) else {
throw HelperError(.pathRejected, "--dest must not contain . or .. segments.")
}
let resolved = URL(fileURLWithPath: directory, isDirectory: true).resolvingSymlinksInPath()
.standardizedFileURL
var isDirectory: ObjCBool = false
guard fileManager.fileExists(atPath: resolved.path, isDirectory: &isDirectory), isDirectory.boolValue
else { throw HelperError(.pathRejected, "--dest must be an existing directory.") }
let fileName = sanitizeFileName(rawFileName)
let finalURL = resolved.appendingPathComponent(fileName, isDirectory: false).standardizedFileURL
let temporaryURL = resolved.appendingPathComponent(
".\(fileName).\(temporarySuffix).partial", isDirectory: false
).standardizedFileURL
for candidate in [finalURL, temporaryURL]
where candidate.deletingLastPathComponent().path != resolved.path {
throw HelperError(.pathRejected, "Export path left the destination directory.")
}
// lstat semantics: a dangling symlink at the final path also counts as existing.
if (try? fileManager.attributesOfItem(atPath: finalURL.path)) != nil {
throw HelperError(.destinationExists, "The destination file already exists.")
}
return ExportDestination(
directory: resolved, finalURL: finalURL, temporaryURL: temporaryURL, fileName: fileName)
}
/// One path component with no separators, control characters or leading dots, at most
/// 200 UTF-8 bytes with the extension kept.
public func sanitizeFileName(_ raw: String) -> String {
var cleaned = String(
raw.unicodeScalars.map { scalar -> Character in
if scalar == "/" || scalar == "\\" || scalar == ":" { return "_" }
if CharacterSet.controlCharacters.contains(scalar) { return "_" }
return Character(scalar)
})
while cleaned.hasPrefix(".") || cleaned.hasPrefix(" ") { cleaned.removeFirst() }
cleaned = cleaned.trimmingCharacters(in: .whitespaces)
if cleaned.isEmpty { return "asset" }
let maxBytes = 200
if cleaned.utf8.count <= maxBytes { return cleaned }
let ext = (cleaned as NSString).pathExtension
let suffix = ext.isEmpty || ext.utf8.count > 16 ? "" : ".\(ext)"
var stem = suffix.isEmpty ? cleaned : String(cleaned.dropLast(suffix.count))
while stem.utf8.count + suffix.utf8.count > maxBytes { stem.removeLast() }
return stem + suffix
}

View File

@ -0,0 +1,148 @@
import AVFoundation
import AppKit
import Foundation
import Photos
import PhotosHelperCore
/// Writes one asset into the destination directory: first to a hidden temporary file,
/// then renamed into place after hashing, so a cancelled export never leaves a file
/// that looks complete.
struct Exporter {
func export(
localIdentifier: String,
directory: String,
variant: ExportVariant,
progress: @escaping (Double) -> Void
) async throws -> ExportOutput {
guard
let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil)
.firstObject
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
guard let resource = originalResource(for: asset, variant: variant) else {
throw HelperError(.notFound, "The asset has no exportable resource.")
}
var fileName = resource.originalFilename
if variant == .proxy {
let stem = (fileName as NSString).deletingPathExtension
fileName = stem + (asset.mediaType == .video ? ".mp4" : ".jpg")
}
let destination = try resolveExportDestination(directory: directory, fileName: fileName)
let fileManager = FileManager.default
do {
switch (variant, asset.mediaType) {
case (.original, _):
try await writeResource(resource, to: destination.temporaryURL, progress: progress)
case (.proxy, .video):
try await writeVideoProxy(asset, to: destination.temporaryURL, progress: progress)
case (.proxy, _):
try await writeImageProxy(asset, to: destination.temporaryURL, progress: progress)
}
let digest = try sha256Hex(of: destination.temporaryURL)
// moveItem refuses an existing destination, so nothing is ever overwritten.
try fileManager.moveItem(at: destination.temporaryURL, to: destination.finalURL)
progress(1)
return ExportOutput(
path: destination.finalURL.path, fileName: destination.fileName,
sizeBytes: digest.sizeBytes, sha256: digest.hex)
} catch {
try? fileManager.removeItem(at: destination.temporaryURL)
if let helperError = error as? HelperError { throw helperError }
throw mapExportError(error)
}
}
/// Original bytes through PHAssetResourceManager; downloads from iCloud when the
/// library keeps only optimized copies on this Mac.
private func writeResource(
_ resource: PHAssetResource, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHAssetResourceRequestOptions()
options.isNetworkAccessAllowed = true
options.progressHandler = { value in progress(value) }
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
PHAssetResourceManager.default().writeData(for: resource, toFile: url, options: options) {
error in
if let error { continuation.resume(throwing: error) } else { continuation.resume() }
}
}
}
/// 720p H.264 MP4 of the current version, without source metadata.
private func writeVideoProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHVideoRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.version = .current
options.progressHandler = { value, _, _, _ in progress(value * 0.5) }
let session: AVAssetExportSession = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestExportSession(
forVideo: asset, options: options, exportPreset: AVAssetExportPreset1280x720
) { session, info in
if let session {
continuation.resume(returning: session)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not prepare the video export."))
}
}
}
session.outputURL = url
session.outputFileType = .mp4
session.shouldOptimizeForNetworkUse = true
session.metadata = []
session.metadataItemFilter = .forSharing()
await session.export()
guard session.status == .completed else {
throw session.error ?? HelperError(.exportFailed, "The video export did not finish.")
}
}
/// JPEG of the current version, at most 1280 px on the long edge, without EXIF/GPS.
private func writeImageProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHImageRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.resizeMode = .exact
options.version = .current
options.isSynchronous = false
options.progressHandler = { value, _, _, _ in progress(value) }
// highQualityFormat calls the handler exactly once.
let image: NSImage = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestImage(
for: asset, targetSize: CGSize(width: 1280, height: 1280), contentMode: .aspectFit,
options: options
) { image, info in
if let image {
continuation.resume(returning: image)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not render the photo."))
}
}
}
guard let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil),
let data = NSBitmapImageRep(cgImage: cgImage).representation(
using: .jpeg, properties: [.compressionFactor: 0.85])
else { throw HelperError(.exportFailed, "Could not encode the photo.") }
try data.write(to: url, options: .withoutOverwriting)
}
}
/// iCloud and network failures are retryable; everything else is an export failure.
func mapExportError(_ error: Error) -> HelperError {
if let photosError = error as? PHPhotosError,
photosError.code == .networkAccessRequired || photosError.code == .networkError
{
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
if (error as NSError).domain == NSURLErrorDomain {
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
return HelperError(.exportFailed, "The export failed.")
}

View File

@ -0,0 +1,183 @@
import Foundation
import Photos
import PhotosHelperCore
/// Read-only PhotoKit queries. Nothing here writes to the library.
struct Library {
private let library = PHPhotoLibrary.shared()
/// Images and videos only; audio and unknown media types are not part of Media Studio.
/// PhotoKit sorts by date only; the store places missing dates first, which is what
/// `SortKey` assumes, and `nextPage` orders ties itself.
private func fetchOptions(since: Date?) -> PHFetchOptions {
let options = PHFetchOptions()
options.includeHiddenAssets = false
var predicates = [
NSPredicate(
format: "mediaType == %ld || mediaType == %ld",
PHAssetMediaType.image.rawValue, PHAssetMediaType.video.rawValue)
]
if let since {
predicates.append(NSPredicate(format: "modificationDate >= %@", since as NSDate))
}
options.predicate = NSCompoundPredicate(andPredicateWithSubpredicates: predicates)
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: true)]
return options
}
func list(after cursor: ListCursor?, limit: Int, since: Date?) throws -> ListOutput {
// Captured before enumeration so changes made while listing show up next time.
let changeToken = try encodeToken(library.currentChangeToken)
let result = PHAsset.fetchAssets(with: fetchOptions(since: since))
let page = nextPage(
count: result.count,
key: { index in
let asset = result.object(at: index)
return SortKey(creationDate: asset.creationDate, localIdentifier: asset.localIdentifier)
},
after: cursor,
limit: limit)
let assets = page.indices.map { result.object(at: $0) }
let items = assets.compactMap(makeItem(for:))
let nextCursor = page.hasMore
? assets.last.map {
ListCursor(key: SortKey(creationDate: $0.creationDate, localIdentifier: $0.localIdentifier))
.encoded()
}
: nil
return ListOutput(items: items, nextCursor: nextCursor, changeToken: changeToken)
}
func asset(_ localIdentifier: String) throws -> AssetOutput {
guard let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil).firstObject,
let item = makeItem(for: asset)
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
let resources = PHAssetResource.assetResources(for: asset).map {
AssetResourceInfo(
type: resourceTypeName($0.type), fileName: $0.originalFilename, uti: $0.uniformTypeIdentifier)
}
return AssetOutput(
item: item, modifiedAt: asset.modificationDate, hidden: asset.isHidden, resources: resources)
}
/// Persistent change history (macOS 13). An expired token or missing history asks the
/// caller for a full relisting instead of guessing.
func changes(since encodedToken: String) throws -> ChangesOutput {
guard let token = decodeToken(encodedToken) else { return .fullResync() }
let current = try encodeToken(library.currentChangeToken)
var changed = Set<String>()
var deleted = Set<String>()
do {
let result = try library.fetchPersistentChanges(since: token)
for change in result {
let details = try change.changeDetails(for: PHObjectType.asset)
changed.formUnion(details.insertedLocalIdentifiers)
changed.formUnion(details.updatedLocalIdentifiers)
deleted.formUnion(details.deletedLocalIdentifiers)
if changed.count + deleted.count > Limits.maxChanges { break }
}
} catch let error as PHPhotosError
where error.code == .persistentChangeTokenExpired
|| error.code == .persistentChangeDetailsUnavailable
{
return .fullResync()
} catch {
throw HelperError(.internal, "Could not read the Photos change history.")
}
if changed.count + deleted.count > Limits.maxChanges { return .fullResync() }
changed.subtract(deleted)
var items: [ClientMediaItem] = []
var found = Set<String>()
if !changed.isEmpty {
PHAsset.fetchAssets(withLocalIdentifiers: Array(changed), options: nil).enumerateObjects {
asset, _, _ in
found.insert(asset.localIdentifier)
if let item = makeItem(for: asset) { items.append(item) }
}
}
// Assets that vanished between the history read and the fetch count as deleted.
deleted.formUnion(changed.subtracting(found))
items.sort { $0.externalId < $1.externalId }
return ChangesOutput(
token: current, changed: items, deleted: deleted.sorted(), fullResyncRequired: false)
}
private func encodeToken(_ token: PHPersistentChangeToken) throws -> String {
guard
let data = try? NSKeyedArchiver.archivedData(withRootObject: token, requiringSecureCoding: true)
else { throw HelperError(.internal, "Could not encode the Photos change token.") }
return base64URLEncode(data)
}
private func decodeToken(_ value: String) -> PHPersistentChangeToken? {
guard let data = base64URLDecode(value) else { return nil }
return try? NSKeyedUnarchiver.unarchivedObject(ofClass: PHPersistentChangeToken.self, from: data)
}
}
/// Builds the contract item. Returns nil for media types Media Studio does not handle
/// and for identifiers longer than the contract allows.
func makeItem(for asset: PHAsset) -> ClientMediaItem? {
let kind: ClientMediaItem.Kind
switch asset.mediaType {
case .video: kind = .video
case .image: kind = asset.mediaSubtypes.contains(.photoLive) ? .livePhoto : .photo
default: return nil
}
guard asset.localIdentifier.utf16.count <= Limits.externalId else { return nil }
var albums: [String] = []
PHAssetCollection.fetchAssetCollectionsContaining(asset, with: .album, options: nil)
.enumerateObjects { collection, _, stop in
if let title = collection.localizedTitle { albums.append(title) }
if albums.count >= Limits.albumCount { stop.pointee = true }
}
let coordinate = asset.location?.coordinate
return ClientMediaItem(
externalId: asset.localIdentifier,
kind: kind,
capturedAt: asset.creationDate,
durationSeconds: kind == .video ? asset.duration : nil,
width: asset.pixelWidth,
height: asset.pixelHeight,
latitude: coordinate?.latitude,
longitude: coordinate?.longitude,
albumNames: albums,
isFavorite: asset.isFavorite,
fileName: originalResource(for: asset, variant: .original)?.originalFilename)
}
/// The resource holding the untouched original: the video track for videos, the still
/// for photos and Live Photos (the paired motion clip is not exported).
func originalResource(for asset: PHAsset, variant: ExportVariant) -> PHAssetResource? {
let resources = PHAssetResource.assetResources(for: asset)
let preferred: [PHAssetResourceType]
switch (asset.mediaType, variant) {
case (.video, .original): preferred = [.video, .fullSizeVideo]
case (.video, .proxy): preferred = [.fullSizeVideo, .video]
case (_, .original): preferred = [.photo, .fullSizePhoto]
case (_, .proxy): preferred = [.fullSizePhoto, .photo]
}
for type in preferred {
if let match = resources.first(where: { $0.type == type }) { return match }
}
return nil
}
func resourceTypeName(_ type: PHAssetResourceType) -> String {
switch type {
case .photo: return "photo"
case .video: return "video"
case .audio: return "audio"
case .alternatePhoto: return "alternate_photo"
case .fullSizePhoto: return "full_size_photo"
case .fullSizeVideo: return "full_size_video"
case .adjustmentData: return "adjustment_data"
case .adjustmentBasePhoto: return "adjustment_base_photo"
case .pairedVideo: return "paired_video"
case .fullSizePairedVideo: return "full_size_paired_video"
case .adjustmentBasePairedVideo: return "adjustment_base_paired_video"
case .adjustmentBaseVideo: return "adjustment_base_video"
@unknown default: return "other"
}
}

View File

@ -0,0 +1,103 @@
import Foundation
import Photos
import PhotosHelperCore
/// Entry point. Exactly one JSON document goes to stdout on success; failures write
/// `{"error":{code,message}}` to stderr and exit non-zero (see HelperError.exitCode).
@main
struct PhotosHelper {
static func main() async {
do {
let command = try parseArguments(Array(CommandLine.arguments.dropFirst()))
let output = try await run(command)
FileHandle.standardOutput.write(Data((output + "\n").utf8))
exit(0)
} catch let error as HelperError {
fail(error)
} catch {
fail(HelperError(.internal, "Unexpected failure."))
}
}
static func fail(_ error: HelperError) -> Never {
FileHandle.standardError.write(Data((errorLine(error) + "\n").utf8))
exit(error.exitCode)
}
static func run(_ command: Command) async throws -> String {
switch command {
case .authorize(let prompt):
let status = await authorize(prompt: prompt)
return try encodeJSON(AuthorizationOutput(status: status))
case .list(let cursor, let limit, let since):
try requireAccess()
return try encodeJSON(try Library().list(after: cursor, limit: limit, since: since))
case .asset(let localIdentifier):
try requireAccess()
return try encodeJSON(try Library().asset(localIdentifier))
case .export(let localIdentifier, let destination, let variant):
try requireAccess()
let progress = ProgressReporter()
// Local copies report no progress, so a heartbeat repeats the last value; the app
// treats a silent helper as stalled.
let heartbeat = Task {
while !Task.isCancelled {
try? await Task.sleep(nanoseconds: 15_000_000_000)
if !Task.isCancelled { progress.emit(nil) }
}
}
defer { heartbeat.cancel() }
progress.emit(0)
let result = try await Exporter().export(
localIdentifier: localIdentifier, directory: destination, variant: variant,
progress: { fraction in progress.emit(fraction) })
return try encodeJSON(result)
case .changes(let token):
try requireAccess()
return try encodeJSON(try Library().changes(since: token))
}
}
/// `.readWrite` is the lowest PhotoKit level that can read assets: `.addOnly` cannot
/// enumerate or export anything. The helper never writes to the library.
static func authorize(prompt: Bool) async -> AuthorizationState {
var status = PHPhotoLibrary.authorizationStatus(for: .readWrite)
if prompt && status == .notDetermined {
status = await PHPhotoLibrary.requestAuthorization(for: .readWrite)
}
return state(status)
}
static func state(_ status: PHAuthorizationStatus) -> AuthorizationState {
switch status {
case .authorized: return .authorized
case .limited: return .limited
case .denied: return .denied
case .restricted: return .restricted
case .notDetermined: return .notDetermined
@unknown default: return .denied
}
}
/// Serializes progress lines written from PhotoKit callback queues and the heartbeat.
final class ProgressReporter: @unchecked Sendable {
private let lock = NSLock()
private var last = 0.0
/// nil repeats the last reported value.
func emit(_ fraction: Double?) {
lock.lock()
defer { lock.unlock() }
if let fraction { last = max(last, fraction) }
FileHandle.standardError.write(Data((progressLine(last) + "\n").utf8))
}
}
/// Every command except `authorize` runs only with granted access and never prompts.
static func requireAccess() throws {
switch state(PHPhotoLibrary.authorizationStatus(for: .readWrite)) {
case .authorized, .limited: return
default: throw HelperError(.notAuthorized, "Photos access has not been granted.")
}
}
}

View File

@ -0,0 +1,73 @@
import XCTest
@testable import PhotosHelperCore
final class ArgumentsTests: XCTestCase {
func testAuthorizePromptsUnlessAskedNotTo() throws {
XCTAssertEqual(try parseArguments(["authorize"]), .authorize(prompt: true))
XCTAssertEqual(try parseArguments(["authorize", "--no-prompt"]), .authorize(prompt: false))
}
func testListDefaultsAndOptions() throws {
XCTAssertEqual(try parseArguments(["list"]), .list(cursor: nil, limit: 500, since: nil))
let cursor = ListCursor(key: SortKey(creationMillis: 1_700_000_000_000, localIdentifier: "A/L0/001"))
let parsed = try parseArguments([
"list", "--cursor", cursor.encoded(), "--limit", "25", "--since", "2024-01-02T03:04:05Z",
])
XCTAssertEqual(
parsed,
.list(cursor: cursor, limit: 25, since: Date(timeIntervalSince1970: 1_704_164_645)))
}
func testListRejectsBadValues() {
for arguments in [
["list", "--limit", "0"],
["list", "--limit", "1001"],
["list", "--limit", "ten"],
["list", "--limit"],
["list", "--since", "yesterday"],
["list", "--cursor", "not a cursor"],
["list", "--limit", "5", "--limit", "6"],
["list", "--dest", "/tmp"],
["list", "extra"],
["list", "--unknown"],
] {
XCTAssertThrowsError(try parseArguments(arguments), "\(arguments)") { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
}
}
}
func testAssetNeedsExactlyOneIdentifier() throws {
XCTAssertEqual(try parseArguments(["asset", "ABC/L0/001"]), .asset(localIdentifier: "ABC/L0/001"))
XCTAssertThrowsError(try parseArguments(["asset"]))
XCTAssertThrowsError(try parseArguments(["asset", "a", "b"]))
XCTAssertThrowsError(try parseArguments(["asset", String(repeating: "x", count: 201)]))
XCTAssertThrowsError(try parseArguments(["asset", "bad\nid"]))
}
func testExportVariants() throws {
XCTAssertEqual(
try parseArguments(["export", "ID", "--dest", "/tmp/out"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .original))
XCTAssertEqual(
try parseArguments(["export", "--dest", "/tmp/out", "ID", "--proxy"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .proxy))
XCTAssertThrowsError(try parseArguments(["export", "ID"]))
XCTAssertThrowsError(try parseArguments(["export", "ID", "--dest", "/tmp", "--original", "--proxy"]))
}
func testChangesNeedsToken() throws {
XCTAssertEqual(try parseArguments(["changes", "--token", "abc"]), .changes(token: "abc"))
XCTAssertThrowsError(try parseArguments(["changes"]))
XCTAssertThrowsError(try parseArguments(["changes", "--token", ""]))
}
func testUnknownCommandIsUsageError() {
XCTAssertThrowsError(try parseArguments([])) { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
XCTAssertEqual((error as? HelperError)?.exitCode, 64)
}
XCTAssertThrowsError(try parseArguments(["delete", "ID"]))
}
}

View File

@ -0,0 +1,92 @@
import XCTest
@testable import PhotosHelperCore
final class OutputTests: XCTestCase {
private func object(_ json: String) throws -> [String: Any] {
try XCTUnwrap(JSONSerialization.jsonObject(with: Data(json.utf8)) as? [String: Any])
}
func testItemEncodesEveryContractKeyWithExplicitNulls() throws {
let item = ClientMediaItem(
externalId: "ID/L0/001", kind: .photo, capturedAt: nil, durationSeconds: nil, width: 0,
height: nil, latitude: nil, longitude: nil, albumNames: [], isFavorite: false, fileName: nil)
let json = try object(encodeJSON(item))
XCTAssertEqual(
Set(json.keys),
[
"externalId", "kind", "capturedAt", "durationMs", "width", "height", "fps", "gpsLat",
"gpsLon", "deviceModel", "albumNames", "isFavorite", "fileName",
])
for key in ["capturedAt", "durationMs", "width", "height", "fps", "gpsLat", "gpsLon", "deviceModel", "fileName"] {
XCTAssertTrue(json[key] is NSNull, key)
}
}
func testItemNormalizesValues() throws {
let item = ClientMediaItem(
externalId: "V/L0/001", kind: .livePhoto, capturedAt: Date(timeIntervalSince1970: 1_704_164_645.5),
durationSeconds: 12.3456, width: 1920, height: 1080, latitude: 52.520_08, longitude: 13.404_95,
albumNames: ["Trip", "", String(repeating: "a", count: 300)] + Array(repeating: "x", count: 60),
isFavorite: true, fileName: "IMG_0001.HEIC")
let json = try object(encodeJSON(item))
XCTAssertEqual(json["kind"] as? String, "live_photo")
XCTAssertEqual(json["capturedAt"] as? String, "2024-01-02T03:04:05.500Z")
XCTAssertEqual(json["durationMs"] as? Int, 12_346)
XCTAssertEqual(json["gpsLat"] as? Double, 52.52)
XCTAssertEqual(json["gpsLon"] as? Double, 13.4)
let albums = try XCTUnwrap(json["albumNames"] as? [String])
XCTAssertEqual(albums.count, 50)
XCTAssertEqual(albums[1].count, 200)
}
func testInvalidCoordinatesAreDropped() {
XCTAssertNil(roundedCoordinate(latitude: 91, longitude: 0.5))
XCTAssertNil(roundedCoordinate(latitude: .nan, longitude: 1))
XCTAssertNil(roundedCoordinate(latitude: 0, longitude: 0))
XCTAssertNil(roundedCoordinate(latitude: 1, longitude: nil))
}
func testClipCountsUTF16Units() {
XCTAssertEqual(clip("abc", maxCharacters: 5), "abc")
// Each emoji is two UTF-16 units; a half character is never kept.
XCTAssertEqual(clip("😀😀😀", maxCharacters: 5), "😀😀")
}
func testErrorAndProgressLinesAreSingleLineJSON() throws {
let line = errorLine(HelperError(.notFound, "No \"asset\"\nhere."))
XCTAssertFalse(line.contains("\n"))
let error = try XCTUnwrap(try object(line)["error"] as? [String: String])
XCTAssertEqual(error["code"], "not_found")
XCTAssertEqual(error["message"], "No \"asset\"\nhere.")
for (input, expected) in [(0.42424, 0.424), (7, 1), (-1, 0), (.nan, 0)] as [(Double, Double)] {
let progress = progressLine(input)
XCTAssertFalse(progress.contains("\n"))
XCTAssertEqual(try object(progress)["progress"] as? Double ?? -1, expected, accuracy: 1e-9)
}
}
func testListAndChangesKeepNullKeys() throws {
let list = try object(encodeJSON(ListOutput(items: [], nextCursor: nil, changeToken: nil)))
XCTAssertTrue(list["nextCursor"] is NSNull)
XCTAssertTrue(list["changeToken"] is NSNull)
let changes = try object(encodeJSON(ChangesOutput.fullResync()))
XCTAssertTrue(changes["token"] is NSNull)
XCTAssertEqual(changes["fullResyncRequired"] as? Bool, true)
}
func testExitCodesAreDistinctPerFailureClass() {
let codes = HelperError.Code.allCases.map { HelperError($0, "").exitCode }
XCTAssertFalse(codes.contains(0))
XCTAssertEqual(HelperError(.notAuthorized, "").exitCode, 77)
}
func testSHA256StreamsFiles() throws {
let url = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
defer { try? FileManager.default.removeItem(at: url) }
try Data("abc".utf8).write(to: url)
let digest = try sha256Hex(of: url)
XCTAssertEqual(digest.hex, "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
XCTAssertEqual(digest.sizeBytes, 3)
}
}

View File

@ -0,0 +1,102 @@
import XCTest
@testable import PhotosHelperCore
final class PagingTests: XCTestCase {
func testCursorRoundTripsAndIsURLSafe() throws {
for key in [
SortKey(creationMillis: 1_700_000_000_123, localIdentifier: "9F2A/L0/001"),
SortKey(creationMillis: nil, localIdentifier: "undated+/=?"),
] {
let encoded = ListCursor(key: key).encoded()
XCTAssertFalse(encoded.contains { "+/=".contains($0) }, encoded)
XCTAssertEqual(try ListCursor.decode(encoded).key, key)
}
}
func testCursorRejectsGarbage() {
for value in ["", "!!!", base64URLEncode(Data("{\"v\":2,\"d\":1,\"id\":\"x\"}".utf8)),
base64URLEncode(Data("{\"v\":1,\"d\":1,\"id\":\"\"}".utf8)), base64URLEncode(Data("[]".utf8))]
{
XCTAssertThrowsError(try ListCursor.decode(value), value)
}
}
func testSortKeyPutsUndatedFirstAndBreaksTiesByIdentifier() {
let keys = [
SortKey(creationMillis: 2, localIdentifier: "a"),
SortKey(creationMillis: 1, localIdentifier: "b"),
SortKey(creationMillis: nil, localIdentifier: "z"),
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: nil, localIdentifier: "c"),
]
XCTAssertEqual(
keys.sorted().map { key -> String in
let date = key.creationMillis.map { millis in String(millis) } ?? "-"
return date + key.localIdentifier
},
["-c", "-z", "1a", "1b", "2a"])
}
/// Simulates PhotoKit: sorted by date, ties in arbitrary (here reversed) order.
private func photoKitOrder(_ keys: [SortKey]) -> [SortKey] {
keys.sorted { lhs, rhs in
if lhs.creationMillis == rhs.creationMillis { return lhs.localIdentifier > rhs.localIdentifier }
return lhs < rhs
}
}
private func listAll(_ source: [SortKey], limit: Int) -> [SortKey] {
var cursor: ListCursor?
var listed: [SortKey] = []
var pages = 0
while true {
let page = nextPage(count: source.count, key: { source[$0] }, after: cursor, limit: limit)
XCTAssertLessThanOrEqual(page.indices.count, limit)
listed += page.indices.map { source[$0] }
pages += 1
guard page.hasMore, let last = listed.last else { break }
// The cursor survives a round trip through its encoded form, like the app stores it.
cursor = try? ListCursor.decode(ListCursor(key: last).encoded())
if pages > source.count + 1 { XCTFail("paging did not terminate"); break }
}
return listed
}
func testPagesCoverEverythingOnceInStableOrder() {
var keys: [SortKey] = []
for index in 0..<57 {
// Many equal timestamps (bursts) and a few undated assets.
let millis: Int64? = index % 11 == 0 ? nil : Int64(index / 4)
keys.append(SortKey(creationMillis: millis, localIdentifier: String(format: "id-%03d", index)))
}
let source = photoKitOrder(keys)
for limit in [1, 3, 4, 10, 500] {
XCTAssertEqual(listAll(source, limit: limit), keys.sorted(), "limit \(limit)")
}
}
func testEmptyLibraryHasNoMorePages() {
let page = nextPage(count: 0, key: { _ in fatalError() }, after: nil, limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testCursorAfterLastItemReturnsNothing() {
let source = [SortKey(creationMillis: 1, localIdentifier: "a")]
let page = nextPage(
count: 1, key: { source[$0] }, after: ListCursor(key: source[0]), limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testDeletedCursorItemStillResumesAfterIt() {
let source = [
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: 3, localIdentifier: "c"),
]
let gone = ListCursor(key: SortKey(creationMillis: 2, localIdentifier: "b"))
let page = nextPage(count: 2, key: { source[$0] }, after: gone, limit: 10)
XCTAssertEqual(page.indices, [1])
}
}

View File

@ -0,0 +1,83 @@
import XCTest
@testable import PhotosHelperCore
final class PathGuardTests: XCTestCase {
private var directory: URL!
override func setUpWithError() throws {
directory = FileManager.default.temporaryDirectory.appendingPathComponent(
"photos-helper-\(UUID().uuidString)", isDirectory: true)
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
}
override func tearDownWithError() throws {
try? FileManager.default.removeItem(at: directory)
}
private func resolved() -> URL { directory.resolvingSymlinksInPath().standardizedFileURL }
func testWritesDirectChildrenOfTheResolvedDirectory() throws {
let target = try resolveExportDestination(
directory: directory.path, fileName: "IMG_0001.MOV", temporarySuffix: "t")
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path)
XCTAssertEqual(target.finalURL.lastPathComponent, "IMG_0001.MOV")
XCTAssertEqual(target.temporaryURL.lastPathComponent, ".IMG_0001.MOV.t.partial")
}
func testRejectsRelativeAndTraversingDirectories() {
for path in ["relative/dir", "\(directory.path)/../x", "\(directory.path)/./", "/a/../b", ""] {
XCTAssertThrowsError(
try resolveExportDestination(directory: path, fileName: "a.jpg"), path
) { error in
XCTAssertEqual((error as? HelperError)?.code, .pathRejected)
}
}
}
func testRejectsMissingDirectoryAndFiles() throws {
XCTAssertThrowsError(
try resolveExportDestination(directory: directory.path + "/missing", fileName: "a.jpg"))
let file = directory.appendingPathComponent("file")
try Data().write(to: file)
XCTAssertThrowsError(try resolveExportDestination(directory: file.path, fileName: "a.jpg"))
}
func testHostileFileNamesStayInside() throws {
for name in ["../../etc/passwd", "/etc/passwd", "..", ".hidden", "a/b\\c:d", "\u{0}x", " "] {
let target = try resolveExportDestination(directory: directory.path, fileName: name)
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path, name)
XCTAssertFalse(target.fileName.hasPrefix("."), name)
XCTAssertFalse(target.fileName.contains("/"), name)
}
XCTAssertEqual(sanitizeFileName("../../etc/passwd"), "_.._etc_passwd")
XCTAssertEqual(sanitizeFileName(""), "asset")
}
func testLongNamesKeepTheirExtension() {
let name = sanitizeFileName(String(repeating: "é", count: 300) + ".mov")
XCTAssertLessThanOrEqual(name.utf8.count, 200)
XCTAssertTrue(name.hasSuffix(".mov"))
}
func testNeverOverwritesAnExistingFileOrSymlink() throws {
try Data().write(to: directory.appendingPathComponent("taken.jpg"))
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "taken.jpg")) {
error in XCTAssertEqual((error as? HelperError)?.code, .destinationExists)
}
try FileManager.default.createSymbolicLink(
atPath: directory.appendingPathComponent("link.jpg").path, withDestinationPath: "/nonexistent")
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "link.jpg"))
}
func testSymlinkedDirectoryResolvesToItsTarget() throws {
let link = directory.appendingPathComponent("link", isDirectory: true)
let real = directory.appendingPathComponent("real", isDirectory: true)
try FileManager.default.createDirectory(at: real, withIntermediateDirectories: false)
try FileManager.default.createSymbolicLink(at: link, withDestinationURL: real)
let target = try resolveExportDestination(directory: link.path, fileName: "a.jpg")
XCTAssertEqual(
target.finalURL.deletingLastPathComponent().path,
real.resolvingSymlinksInPath().standardizedFileURL.path)
}
}

View File

@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!-- electron-builder's default hardened-runtime entitlements for Electron. -->
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<!-- Hardened runtime blocks Photos without this, for the app and the bundled photos-helper. -->
<key>com.apple.security.personal-information.photos-library</key>
<true/>
</dict>
</plist>

View File

@ -16,6 +16,8 @@
"release": "pnpm --filter @rakazo/web build && pnpm build && electron-builder --publish never"
},
"dependencies": {
"@orpc/client": "1.15.0",
"@orpc/contract": "1.15.0",
"@rakazo/contracts": "workspace:*",
"electron-updater": "6.8.9"
},
@ -61,7 +63,18 @@
"dmg",
"zip"
],
"notarize": true
"notarize": true,
"entitlements": "native/photos-helper/entitlements.mac.plist",
"entitlementsInherit": "native/photos-helper/entitlements.mac.plist",
"extendInfo": {
"NSPhotoLibraryUsageDescription": "Rakazo adds your photos and videos to the media library on your Rakazo server."
},
"extraResources": [
{
"from": "native/photos-helper/.build/apple/Products/Release/photos-helper",
"to": "bin/photos-helper"
}
]
},
"win": {
"icon": "assets/icon.ico",

View File

@ -2,7 +2,10 @@ import { randomBytes } from "node:crypto";
import { existsSync } from "node:fs";
import { readFile, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import type { DesktopReachability, DesktopSetup } from "@rakazo/contracts";
import { createORPCClient } from "@orpc/client";
import { RPCLink } from "@orpc/client/fetch";
import type { ContractRouterClient } from "@orpc/contract";
import type { AppContract, DesktopReachability, DesktopSetup } from "@rakazo/contracts";
import { LOCAL_SETTINGS_PAGE } from "@rakazo/contracts/local-settings";
import {
app,
@ -32,6 +35,14 @@ import {
stackDir,
stackResourceDir,
} from "./local-stack.js";
import type { MediaIpcEvent } from "./media-ipc.js";
import { registerMediaIpc } from "./media-ipc.js";
import { createPhotosHelper, resolvePhotosHelperBinary } from "./media-photos-helper.js";
import {
createChunkUploader,
createPhotosSyncFileStore,
createPhotosSyncRunner,
} from "./media-photos-sync.js";
import { oauthCallbackFrom } from "./oauth-callback.js";
import {
bundledRendererCandidates,
@ -1130,6 +1141,60 @@ app.whenReady().then(async () => {
);
},
);
const photosBinary = resolvePhotosHelperBinary({
platform: process.platform,
packaged: app.isPackaged,
resourcesPath: process.resourcesPath,
appPath: app.getAppPath(),
exists: existsSync,
});
const photosHelper = photosBinary ? createPhotosHelper(photosBinary) : null;
const mediaIpc = registerMediaIpc({
ipcMain,
platform: process.platform,
helper: photosHelper,
isTrusted: (raw: MediaIpcEvent) => {
const event = raw as Electron.IpcMainInvokeEvent;
return fromMainWindow(event) && event.senderFrame === event.sender.mainFrame;
},
runSync: photosHelper
? createPhotosSyncRunner({
helper: photosHelper,
store: createPhotosSyncFileStore(userDataDir),
tempRoot: app.getPath("temp"),
label: "Apple Photos",
connect: async (spaceId) => {
if (currentTargetUrl === null) throw new Error("No server is open.");
const origin = new URL(currentTargetUrl).origin;
const target = (await resolveSessionForTarget(currentTargetUrl)).value;
const fetchWithSession = (input: string | Request, init?: RequestInit) =>
target.fetch(input, {
...init,
credentials: "include",
bypassCustomProtocolHandlers: true,
});
const api: ContractRouterClient<AppContract> = createORPCClient(
new RPCLink({
url: `${origin}/rpc`,
fetch: (request, init) => {
const headers = new Headers(request.headers);
if (spaceId) headers.set("x-rakazo-space-id", spaceId);
return fetchWithSession(new Request(request, { headers }), init);
},
}),
);
return {
origin,
api,
uploadChunk: createChunkUploader({ origin, spaceId, fetch: fetchWithSession }),
};
},
})
: async () => {
throw new Error("Apple Photos is unavailable.");
},
});
app.on("before-quit", mediaIpc.cancel);
ipcMain.handle("desktop.platform", () => process.platform);
const browserPreview = new BrowserPreviewController();
// The in-app browser answers only the app window's own top frame, never a page inside it.

View File

@ -0,0 +1,161 @@
import { describe, expect, it, vi } from "vitest";
import type { MediaIpcEvent, MediaIpcOptions } from "./media-ipc.js";
import {
MEDIA_PHOTOS_CHANNELS,
outcomeForError,
parseSyncRequest,
registerMediaIpc,
} from "./media-ipc.js";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import type { PhotosSyncResult } from "./media-photos-sync.js";
import { PhotosSyncError } from "./media-photos-sync.js";
type Listener = (event: MediaIpcEvent, ...args: unknown[]) => unknown;
const result: PhotosSyncResult = {
indexed: 2,
created: 2,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: true,
};
function setup(overrides: Partial<MediaIpcOptions> = {}) {
const handlers = new Map<string, Listener>();
const helper = {
authorize: vi.fn(async () => "authorized" as const),
} as unknown as PhotosHelper;
const options: MediaIpcOptions = {
ipcMain: { handle: (channel, listener) => handlers.set(channel, listener) },
platform: "darwin",
helper,
isTrusted: () => true,
runSync: vi.fn(async () => result),
...overrides,
};
const registration = registerMediaIpc(options);
const sent: [string, unknown][] = [];
const event: MediaIpcEvent = {
sender: { send: (channel, payload) => sent.push([channel, payload]), isDestroyed: () => false },
};
const call = (channel: string, ...args: unknown[]) => handlers.get(channel)!(event, ...args);
return { handlers, helper, options, registration, call, sent };
}
describe("media IPC on platforms without Apple Photos", () => {
it.each(["linux", "win32"])(
"reports %s as unavailable and rejects sync and cancel",
async (platform) => {
const { call, options, handlers } = setup({ platform });
expect([...handlers.keys()].sort()).toEqual(
[
MEDIA_PHOTOS_CHANNELS.cancel,
MEDIA_PHOTOS_CHANNELS.status,
MEDIA_PHOTOS_CHANNELS.sync,
].sort(),
);
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: null })).toThrow(/macOS/);
expect(() => call(MEDIA_PHOTOS_CHANNELS.cancel)).toThrow(/macOS/);
expect(options.runSync).not.toHaveBeenCalled();
expect(options.helper!.authorize).not.toHaveBeenCalled();
},
);
it("treats a macOS build without the helper as unavailable", async () => {
const { call } = setup({ helper: null });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow(/macOS/);
});
it("treats a helper that cannot start as unavailable", async () => {
const { call, helper } = setup();
vi.mocked(helper.authorize).mockRejectedValueOnce(new PhotosHelperError("unavailable", "x"));
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
});
});
describe("media IPC on macOS", () => {
it("reads status without prompting", async () => {
const { call, helper } = setup();
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({
available: true,
authorization: "authorized",
syncing: false,
});
expect(helper.authorize).toHaveBeenCalledWith(false);
});
it("runs one sync at a time, forwards progress and resolves with the outcome", async () => {
let finish: (value: PhotosSyncResult) => void = () => {};
const runSync = vi.fn(
({ onProgress }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((resolve) => {
onProgress({ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 });
finish = resolve;
}),
);
const { call, sent } = setup({ runSync });
const first = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
const second = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
expect(second).toBe(first);
expect(runSync).toHaveBeenCalledTimes(1);
expect(runSync.mock.calls[0]?.[0].spaceId).toBe("space_1");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: true });
expect(sent).toEqual([
[
MEDIA_PHOTOS_CHANNELS.progress,
{ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 },
],
]);
finish(result);
await expect(first).resolves.toEqual({ ok: true, result });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: false });
});
it("cancels the running sync", async () => {
const runSync = vi.fn(
({ signal }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((_resolve, reject) => {
signal.addEventListener("abort", () =>
reject(new PhotosSyncError("cancelled", "Sync cancelled.")),
);
}),
);
const { call } = setup({ runSync });
const running = call(MEDIA_PHOTOS_CHANNELS.sync, {});
await call(MEDIA_PHOTOS_CHANNELS.cancel);
await expect(running).resolves.toEqual({
ok: false,
code: "cancelled",
message: "Sync cancelled.",
});
});
it("rejects untrusted senders and malformed requests", async () => {
const { call } = setup({ isTrusted: () => false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow("Not allowed.");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).rejects.toThrow("Not allowed.");
expect(() => parseSyncRequest(null)).toThrow();
expect(() => parseSyncRequest({ spaceId: "../x" })).toThrow();
expect(() => parseSyncRequest({ spaceId: 4 })).toThrow();
expect(parseSyncRequest({})).toEqual({ spaceId: null });
});
it("never forwards unexpected error text to the renderer", () => {
expect(outcomeForError(new Error("/Users/example/secret path"))).toEqual({
ok: false,
code: "failed",
message: "Apple Photos sync failed.",
});
expect(
outcomeForError(
new PhotosHelperError("not_authorized", "Photos access has not been granted."),
),
).toMatchObject({
code: "not_authorized",
});
});
});

View File

@ -0,0 +1,126 @@
import type {
PhotosSourceStatus,
PhotosSyncOutcome,
PhotosSyncProgress,
PhotosSyncResult,
RakazoDesktopMediaPhotos,
} from "@rakazo/contracts";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import { PhotosSyncError } from "./media-photos-sync.js";
/*
* IPC for the Apple Photos source. The renderer only expresses intent (sync, cancel)
* and renders state; the main process owns the helper, the session and the upload.
*/
export const MEDIA_PHOTOS_CHANNELS = {
status: "desktop.media.photos.status",
sync: "desktop.media.photos.sync",
cancel: "desktop.media.photos.cancel",
progress: "desktop.media.photos.progress",
} as const;
export type { PhotosSourceStatus, PhotosSyncOutcome, RakazoDesktopMediaPhotos };
export interface MediaIpcEvent {
sender: { send(channel: string, payload: unknown): void; isDestroyed(): boolean };
}
export interface MediaIpcMain {
handle(channel: string, listener: (event: MediaIpcEvent, ...args: unknown[]) => unknown): void;
}
export interface MediaIpcOptions {
ipcMain: MediaIpcMain;
platform: string;
/** Null when the helper binary is missing (unbuilt dev checkout). */
helper: PhotosHelper | null;
/** Same gate as the other app-window handlers: main window, main frame. */
isTrusted: (event: MediaIpcEvent) => boolean;
runSync: (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}) => Promise<PhotosSyncResult>;
}
const UNAVAILABLE = "Apple Photos is only available in the macOS app.";
const SPACE_ID = /^[A-Za-z0-9_-]{1,200}$/;
export function parseSyncRequest(value: unknown): { spaceId: string | null } {
if (typeof value !== "object" || value === null) throw new Error("Invalid sync request.");
const spaceId = (value as { spaceId?: unknown }).spaceId ?? null;
if (spaceId !== null && (typeof spaceId !== "string" || !SPACE_ID.test(spaceId))) {
throw new Error("Invalid sync request.");
}
return { spaceId };
}
/** Registers the handlers; `cancel()` stops a running sync, e.g. before quit. */
export function registerMediaIpc(options: MediaIpcOptions): { cancel: () => void } {
const available = options.platform === "darwin" && options.helper !== null;
let running: { controller: AbortController; done: Promise<PhotosSyncOutcome> } | null = null;
const requireTrusted = (event: MediaIpcEvent) => {
if (!options.isTrusted(event)) throw new Error("Not allowed.");
};
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.status, async (event) => {
requireTrusted(event);
if (!available) return { available: false } satisfies PhotosSourceStatus;
try {
const authorization = await options.helper!.authorize(false);
return { available: true, authorization, syncing: running !== null };
} catch (error) {
if (error instanceof PhotosHelperError && error.code === "unavailable") {
return { available: false } satisfies PhotosSourceStatus;
}
throw error;
}
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.sync, (event, request: unknown) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
const { spaceId } = parseSyncRequest(request);
// A second click joins the running sync instead of starting a parallel one.
if (running) return running.done;
const controller = new AbortController();
const done = options
.runSync({
spaceId,
signal: controller.signal,
onProgress: (progress) => {
if (!event.sender.isDestroyed()) {
event.sender.send(MEDIA_PHOTOS_CHANNELS.progress, progress);
}
},
})
.then(
(result): PhotosSyncOutcome => ({ ok: true, result }),
(error: unknown): PhotosSyncOutcome => outcomeForError(error),
)
.finally(() => {
if (running?.controller === controller) running = null;
});
running = { controller, done };
return done;
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.cancel, (event) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
running?.controller.abort();
});
return { cancel: () => running?.controller.abort() };
}
/** Only our own fixed messages reach the renderer; anything else is generic. */
export function outcomeForError(error: unknown): PhotosSyncOutcome {
if (error instanceof PhotosSyncError || error instanceof PhotosHelperError) {
return { ok: false, code: error.code, message: error.message };
}
return { ok: false, code: "failed", message: "Apple Photos sync failed." };
}

View File

@ -0,0 +1,437 @@
import { EventEmitter } from "node:events";
import { mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { PassThrough } from "node:stream";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { HelperProcess, SpawnHelper } from "./media-photos-helper.js";
import {
changesArgs,
createPhotosHelper,
exportArgs,
helperEnv,
listArgs,
PhotosHelperError,
parseAssetOutput,
parseAuthorizeOutput,
parseChangesOutput,
parseExportOutput,
parseHelperError,
parseListOutput,
parseProgressLine,
photosHelperCandidates,
resolvePhotosHelperBinary,
runPhotosHelper,
} from "./media-photos-helper.js";
const item = (overrides: Record<string, unknown> = {}) => ({
externalId: "9F2A61B0-0000-4000-8000-000000000001/L0/001",
kind: "video",
capturedAt: "2024-01-02T03:04:05.500Z",
durationMs: 12_346,
width: 1920,
height: 1080,
fps: null,
gpsLat: 52.52,
gpsLon: 13.4,
deviceModel: null,
albumNames: ["Trip"],
isFavorite: true,
fileName: "IMG_0001.MOV",
...overrides,
});
const listJson = (items: unknown[], nextCursor: unknown = null, changeToken: unknown = "tok") =>
JSON.stringify({ items, nextCursor, changeToken });
function expectCode(run: () => unknown, code: string) {
try {
run();
} catch (error) {
expect(error).toBeInstanceOf(PhotosHelperError);
expect((error as PhotosHelperError).code).toBe(code);
return;
}
throw new Error("expected a PhotosHelperError");
}
describe("photos helper output parsing", () => {
it("accepts a list page that matches the shared item contract", () => {
const page = parseListOutput(
listJson([item(), item({ kind: "photo", durationMs: null })], "c1"),
2,
);
expect(page.nextCursor).toBe("c1");
expect(page.changeToken).toBe("tok");
expect(page.items).toHaveLength(2);
expect(page.items[0]).toEqual(item());
});
it("accepts the last page of an empty library", () => {
expect(parseListOutput(listJson([], null, null), 500)).toEqual({
items: [],
nextCursor: null,
changeToken: null,
});
});
it.each([
["not JSON", "items: []"],
["an array document", "[]"],
["truncated JSON", '{"items":[{"externalId":'],
["missing items", JSON.stringify({ nextCursor: null, changeToken: null })],
["an item missing a key", listJson([{ ...item(), width: undefined }])],
["an unknown kind", listJson([item({ kind: "audio" })])],
["out-of-range GPS", listJson([item({ gpsLat: 91 })])],
["a non-UTC-parsable date", listJson([item({ capturedAt: "yesterday" })])],
["a negative duration", listJson([item({ durationMs: -1 })])],
["an empty external id", listJson([item({ externalId: "" })])],
["an empty cursor", listJson([item()], "")],
["a numeric cursor", listJson([item()], 3)],
["too many items", listJson([item(), item(), item()])],
])("rejects %s", (_name, text) => {
expectCode(() => parseListOutput(text, 2), "invalid_output");
});
it("parses authorization states and rejects unknown ones", () => {
expect(parseAuthorizeOutput('{"status":"limited"}')).toBe("limited");
expect(parseAuthorizeOutput('{"status":"notDetermined"}')).toBe("notDetermined");
expectCode(() => parseAuthorizeOutput('{"status":"maybe"}'), "invalid_output");
expectCode(() => parseAuthorizeOutput(""), "invalid_output");
});
it("parses asset details", () => {
const detail = parseAssetOutput(
JSON.stringify({
item: item(),
modifiedAt: "2024-02-01T00:00:00.000Z",
hidden: false,
resources: [{ type: "video", fileName: "IMG_0001.MOV", uti: "com.apple.quicktime-movie" }],
}),
);
expect(detail.resources[0]?.type).toBe("video");
expectCode(
() => parseAssetOutput(JSON.stringify({ item: item(), modifiedAt: null, resources: [] })),
"invalid_output",
);
});
it("parses changes and requires a token unless a full relisting is needed", () => {
expect(
parseChangesOutput(
JSON.stringify({
token: "t2",
changed: [item()],
deleted: ["x/L0/001"],
fullResyncRequired: false,
}),
),
).toMatchObject({ token: "t2", deleted: ["x/L0/001"], fullResyncRequired: false });
expect(
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: true }),
).fullResyncRequired,
).toBe(true);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: false }),
),
"invalid_output",
);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: "t", changed: [], deleted: [7], fullResyncRequired: false }),
),
"invalid_output",
);
});
it("maps helper error lines to codes", () => {
const stderr =
'{"progress":0.5}\n{"error":{"code":"not_authorized","message":"Photos access has not been granted."}}\n';
const error = parseHelperError(stderr, 77);
expect(error.code).toBe("not_authorized");
expect(error.message).toBe("Photos access has not been granted.");
expect(parseHelperError('{"error":{"code":"internal","message":"x"}}', 70).code).toBe("failed");
expect(parseHelperError("Segmentation fault", 139).message).toContain("139");
expect(parseHelperError('{"error":"oops"}', 1).code).toBe("failed");
});
it("reads progress lines only", () => {
expect(parseProgressLine('{"progress":0.42}')).toBe(0.42);
expect(parseProgressLine('{"progress":2}')).toBeNull();
expect(parseProgressLine('{"error":{"code":"usage"}}')).toBeNull();
expect(parseProgressLine('{"progress":')).toBeNull();
});
});
describe("photos helper export result", () => {
let dir: string;
let other: string;
beforeEach(async () => {
dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-export-"));
other = await mkdtemp(path.join(tmpdir(), "rakazo-photos-other-"));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
await rm(other, { recursive: true, force: true });
});
const result = (file: string, fileName = path.basename(file)) =>
JSON.stringify({ path: file, fileName, sizeBytes: 3, sha256: "a".repeat(64) });
it("accepts a file directly inside the export folder", async () => {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
await expect(parseExportOutput(result(file), dir)).resolves.toMatchObject({
fileName: "IMG_0001.MOV",
sizeBytes: 3,
});
});
it("rejects files outside the export folder, including through symlinks", async () => {
const outside = path.join(other, "secret.txt");
await writeFile(outside, "abc");
await expect(parseExportOutput(result(outside), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const link = path.join(dir, "link.MOV");
await symlink(outside, link);
await expect(parseExportOutput(result(link), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const nested = path.join(dir, "..", path.basename(other), "secret.txt");
await expect(parseExportOutput(result(nested), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
it("rejects malformed results", async () => {
const file = path.join(dir, "a.jpg");
await writeFile(file, "abc");
for (const text of [
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: 3, sha256: "XYZ" }),
JSON.stringify({ path: "a.jpg", fileName: "a.jpg", sizeBytes: 3, sha256: "a".repeat(64) }),
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: -1, sha256: "a".repeat(64) }),
JSON.stringify({
path: path.join(dir, "missing.jpg"),
fileName: "missing.jpg",
sizeBytes: 3,
sha256: "a".repeat(64),
}),
]) {
await expect(parseExportOutput(text, dir)).rejects.toMatchObject({ code: "invalid_output" });
}
await expect(parseExportOutput(result(file, "b.jpg"), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
});
describe("photos helper argv", () => {
it("builds fixed argv arrays", () => {
expect(listArgs({ cursor: null, limit: 500 })).toEqual(["list", "--limit", "500"]);
expect(listArgs({ cursor: "abc_-", limit: 10, since: "2024-01-01T00:00:00Z" })).toEqual([
"list",
"--limit",
"10",
"--cursor",
"abc_-",
"--since",
"2024-01-01T00:00:00Z",
]);
expect(exportArgs("ID/L0/001", "/tmp/rakazo-photos-1")).toEqual([
"export",
"ID/L0/001",
"--dest",
"/tmp/rakazo-photos-1",
"--original",
]);
expect(changesArgs("tok")).toEqual(["changes", "--token", "tok"]);
});
it("keeps shell syntax as inert data and rejects option-like values", () => {
// No shell runs, so metacharacters stay one literal argument.
expect(exportArgs("$(rm -rf ~);`x`", "/tmp/a b")[1]).toBe("$(rm -rf ~);`x`");
expectCode(() => exportArgs("--dest", "/tmp"), "usage");
expectCode(() => listArgs({ cursor: "--limit", limit: 1 }), "usage");
expectCode(() => changesArgs("a\nb"), "usage");
expectCode(() => listArgs({ cursor: null, limit: 501 }), "usage");
expectCode(() => exportArgs("ID", "relative/dir"), "path_rejected");
expectCode(() => exportArgs("ID", "/tmp/../etc"), "path_rejected");
});
it("passes only the environment PhotoKit needs", () => {
expect(
helperEnv({ HOME: "/Users/example", TMPDIR: "/tmp", OPENAI_API_KEY: "secret", PATH: "/bin" }),
).toEqual({ HOME: "/Users/example", TMPDIR: "/tmp" });
});
});
describe("photos helper binary", () => {
it("is only available on macOS", () => {
const options = {
packaged: true,
resourcesPath: "/App/Resources",
appPath: "/app",
exists: () => true,
};
expect(resolvePhotosHelperBinary({ ...options, platform: "linux" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "win32" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "darwin" })).toBe(
"/App/Resources/bin/photos-helper",
);
});
it("falls back to the development build and reports a missing build", () => {
const options = {
platform: "darwin",
packaged: false,
resourcesPath: "/r",
appPath: "/repo/apps/desktop",
};
expect(photosHelperCandidates(options)).toEqual([
"/repo/apps/desktop/native/photos-helper/.build/apple/Products/Release/photos-helper",
"/repo/apps/desktop/native/photos-helper/.build/release/photos-helper",
]);
expect(
resolvePhotosHelperBinary({
...options,
exists: (file) => file.endsWith(".build/release/photos-helper"),
}),
).toBe("/repo/apps/desktop/native/photos-helper/.build/release/photos-helper");
expect(resolvePhotosHelperBinary({ ...options, exists: () => false })).toBeNull();
});
});
class FakeProcess extends EventEmitter implements HelperProcess {
stdout = new PassThrough();
stderr = new PassThrough();
kill = vi.fn(() => {
queueMicrotask(() => this.emit("close", null));
return true;
});
exit(code: number, stdout = "", stderr = "") {
if (stdout) this.stdout.write(stdout);
if (stderr) this.stderr.write(stderr);
setImmediate(() => this.emit("close", code));
}
}
function fakeSpawn() {
const processes: FakeProcess[] = [];
const spawn = vi.fn<SpawnHelper>(() => {
const child = new FakeProcess();
processes.push(child);
return child;
});
return { spawn, processes };
}
describe("photos helper process", () => {
it("spawns the binary with argv and no shell", async () => {
const { spawn, processes } = fakeSpawn();
const helper = createPhotosHelper("/App/Resources/bin/photos-helper", spawn);
const pending = helper.list({ cursor: null, limit: 2 });
processes[0]!.exit(0, listJson([item()]));
await expect(pending).resolves.toMatchObject({ items: [item()] });
expect(spawn).toHaveBeenCalledWith(
"/App/Resources/bin/photos-helper",
["list", "--limit", "2"],
expect.objectContaining({ shell: false, stdio: ["ignore", "pipe", "pipe"] }),
);
});
it("maps a failing exit to the helper's error code", async () => {
const { spawn, processes } = fakeSpawn();
const pending = createPhotosHelper("/bin/photos-helper", spawn).asset("ID/L0/001");
processes[0]!.exit(
66,
"",
'{"error":{"code":"not_found","message":"No photo or video with that identifier."}}\n',
);
await expect(pending).rejects.toMatchObject({ code: "not_found" });
});
it("reports export progress and validates the result", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-run-"));
try {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
const { spawn, processes } = fakeSpawn();
const progress: number[] = [];
const pending = createPhotosHelper("/bin/photos-helper", spawn).exportOriginal("ID", dir, {
onProgress: (value) => progress.push(value),
});
const child = processes[0]!;
child.stderr.write('{"progress":0}\n{"progress":0.5');
child.stderr.write('}\nnoise\n{"progress":1}\n');
child.exit(
0,
JSON.stringify({
path: file,
fileName: "IMG_0001.MOV",
sizeBytes: 3,
sha256: "b".repeat(64),
}),
);
await expect(pending).resolves.toMatchObject({ sizeBytes: 3 });
expect(progress).toEqual([0, 0.5, 1]);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("kills the helper on abort and on timeout", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
const aborted = runPhotosHelper("/bin/h", ["list"], {
spawn,
timeoutMs: 60_000,
signal: controller.signal,
});
controller.abort();
await expect(aborted).rejects.toMatchObject({ code: "cancelled" });
expect(processes[0]!.kill).toHaveBeenCalledWith("SIGTERM");
const timedOut = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 5 });
await expect(timedOut).rejects.toMatchObject({ code: "timeout" });
expect(processes[1]!.kill).toHaveBeenCalled();
});
it("fails a silent export through the idle limit", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["export"], {
spawn,
timeoutMs: 60_000,
idleTimeoutMs: 5,
});
await expect(pending).rejects.toMatchObject({ code: "timeout" });
expect(processes[0]!.kill).toHaveBeenCalled();
});
it("does not start when already aborted and reports spawn failures", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
controller.abort();
await expect(
runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000, signal: controller.signal }),
).rejects.toMatchObject({ code: "cancelled" });
expect(spawn).not.toHaveBeenCalled();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.emit("error", new Error("ENOENT"));
await expect(pending).rejects.toMatchObject({ code: "unavailable" });
});
it("rejects oversized output", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.stdout.write(Buffer.alloc(33 * 1024 * 1024));
await expect(pending).rejects.toMatchObject({ code: "invalid_output" });
});
});

View File

@ -0,0 +1,512 @@
import { spawn as nodeSpawn } from "node:child_process";
import { realpath } from "node:fs/promises";
import path from "node:path";
import type { Readable } from "node:stream";
import type { ClientMediaItem, PhotosAuthorization } from "@rakazo/contracts";
import { ClientMediaItemSchema } from "@rakazo/contracts";
/*
* Client for the macOS `photos-helper` CLI (apps/desktop/native/photos-helper). The
* helper runs with a fixed argv and no shell; every byte it prints is validated before
* the app uses it, so a malformed or unexpected answer fails instead of being guessed at.
*/
export type { PhotosAuthorization };
export interface PhotosListPage {
items: ClientMediaItem[];
nextCursor: string | null;
/** Library change token from before this page was read. */
changeToken: string | null;
}
export interface PhotosAssetDetail {
item: ClientMediaItem;
modifiedAt: string | null;
hidden: boolean;
resources: { type: string; fileName: string; uti: string }[];
}
export interface PhotosChanges {
token: string | null;
changed: ClientMediaItem[];
deleted: string[];
fullResyncRequired: boolean;
}
export interface PhotosExport {
path: string;
fileName: string;
sizeBytes: number;
sha256: string;
}
export type PhotosHelperErrorCode =
| "unavailable"
| "usage"
| "not_authorized"
| "not_found"
| "path_rejected"
| "destination_exists"
| "network"
| "export_failed"
| "invalid_output"
| "timeout"
| "cancelled"
| "failed";
const HELPER_CODES = new Set<PhotosHelperErrorCode>([
"usage",
"not_authorized",
"not_found",
"path_rejected",
"destination_exists",
"network",
"export_failed",
]);
export class PhotosHelperError extends Error {
constructor(
readonly code: PhotosHelperErrorCode,
message: string,
) {
super(message);
this.name = "PhotosHelperError";
}
}
export const PHOTOS_HELPER_NAME = "photos-helper";
/** Matches the contract's batch limit for `media.sources.pushItems`. */
export const PHOTOS_MAX_PAGE = 500;
/**
* Packaged apps ship the universal helper in Resources/bin; development uses the
* SwiftPM output (universal `--arch` build first, then the host-arch build).
*/
export function photosHelperCandidates(options: {
packaged: boolean;
resourcesPath: string;
appPath: string;
}): string[] {
if (options.packaged) return [path.join(options.resourcesPath, "bin", PHOTOS_HELPER_NAME)];
const build = path.join(options.appPath, "native", "photos-helper", ".build");
return [
path.join(build, "apple", "Products", "Release", PHOTOS_HELPER_NAME),
path.join(build, "release", PHOTOS_HELPER_NAME),
];
}
/** Null off macOS or when the helper was not built; the Apple Photos source is then hidden. */
export function resolvePhotosHelperBinary(options: {
platform: string;
packaged: boolean;
resourcesPath: string;
appPath: string;
exists: (file: string) => boolean;
}): string | null {
if (options.platform !== "darwin") return null;
return photosHelperCandidates(options).find((candidate) => options.exists(candidate)) ?? null;
}
// ---------------------------------------------------------------------------
// argv
// ---------------------------------------------------------------------------
export function authorizeArgs(prompt: boolean): string[] {
return prompt ? ["authorize"] : ["authorize", "--no-prompt"];
}
export function listArgs(request: {
cursor: string | null;
limit: number;
since?: string | null;
}): string[] {
if (!Number.isInteger(request.limit) || request.limit < 1 || request.limit > PHOTOS_MAX_PAGE) {
throw new PhotosHelperError("usage", "List limit is out of range.");
}
const args = ["list", "--limit", String(request.limit)];
if (request.cursor !== null) args.push("--cursor", opaqueArgument(request.cursor));
if (request.since) args.push("--since", opaqueArgument(request.since));
return args;
}
export function assetArgs(localIdentifier: string): string[] {
return ["asset", identifierArgument(localIdentifier)];
}
export function exportArgs(localIdentifier: string, destination: string): string[] {
if (!path.isAbsolute(destination) || destination.split(/[\\/]/).includes("..")) {
throw new PhotosHelperError("path_rejected", "Export directory must be absolute.");
}
return ["export", identifierArgument(localIdentifier), "--dest", destination, "--original"];
}
export function changesArgs(token: string): string[] {
return ["changes", "--token", opaqueArgument(token)];
}
/** Identifiers come from the helper itself; a leading dash would read as an option. */
function identifierArgument(value: string): string {
if (value === "" || value.length > 200 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid Photos identifier.");
}
return value;
}
function opaqueArgument(value: string): string {
if (value === "" || value.length > 4096 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid helper argument.");
}
return value;
}
function hasControl(value: string): boolean {
for (const char of value) {
const code = char.codePointAt(0) ?? 0;
if (code < 0x20 || code === 0x7f) return true;
}
return false;
}
// ---------------------------------------------------------------------------
// Output parsing
// ---------------------------------------------------------------------------
function invalid(what: string): never {
throw new PhotosHelperError("invalid_output", `Photos helper returned invalid ${what}.`);
}
function parseDocument(text: string): Record<string, unknown> {
let value: unknown;
try {
value = JSON.parse(text);
} catch {
invalid("JSON");
}
if (typeof value !== "object" || value === null || Array.isArray(value)) invalid("JSON");
return value as Record<string, unknown>;
}
function nullableString(value: unknown, what: string, max = 4096): string | null {
if (value === null) return null;
if (typeof value !== "string" || value === "" || value.length > max) invalid(what);
return value;
}
/** Each item must satisfy the shared contract schema exactly as the server will parse it. */
function parseItems(value: unknown, what: string): ClientMediaItem[] {
if (!Array.isArray(value) || value.length > 10_000) invalid(what);
return value.map((entry) => {
if (typeof entry !== "object" || entry === null) invalid(what);
const parsed = ClientMediaItemSchema.safeParse(entry);
if (!parsed.success) invalid(what);
return parsed.data;
});
}
const AUTHORIZATION = new Set<PhotosAuthorization>([
"authorized",
"limited",
"denied",
"restricted",
"notDetermined",
]);
export function parseAuthorizeOutput(text: string): PhotosAuthorization {
const status = parseDocument(text).status;
if (typeof status !== "string" || !AUTHORIZATION.has(status as PhotosAuthorization)) {
invalid("authorization status");
}
return status as PhotosAuthorization;
}
export function parseListOutput(text: string, limit: number): PhotosListPage {
const document = parseDocument(text);
const items = parseItems(document.items, "list items");
if (items.length > limit) invalid("list page size");
const nextCursor = nullableString(document.nextCursor, "list cursor");
const changeToken = nullableString(document.changeToken, "change token");
return { items, nextCursor, changeToken };
}
export function parseAssetOutput(text: string): PhotosAssetDetail {
const document = parseDocument(text);
const [item] = parseItems([document.item], "asset");
const modifiedAt = nullableString(document.modifiedAt, "asset date", 64);
if (modifiedAt !== null && Number.isNaN(Date.parse(modifiedAt))) invalid("asset date");
if (typeof document.hidden !== "boolean") invalid("asset");
if (!Array.isArray(document.resources) || document.resources.length > 50) invalid("resources");
const resources = document.resources.map((resource) => {
if (typeof resource !== "object" || resource === null) invalid("resources");
const { type, fileName, uti } = resource as Record<string, unknown>;
if (typeof type !== "string" || typeof fileName !== "string" || typeof uti !== "string") {
invalid("resources");
}
return { type, fileName, uti };
});
return { item: item!, modifiedAt, hidden: document.hidden, resources };
}
export function parseChangesOutput(text: string): PhotosChanges {
const document = parseDocument(text);
const token = nullableString(document.token, "change token");
if (typeof document.fullResyncRequired !== "boolean") invalid("changes");
const changed = parseItems(document.changed, "changed items");
if (!Array.isArray(document.deleted) || document.deleted.length > 10_000) invalid("changes");
const deleted = document.deleted.map((id) => {
if (typeof id !== "string" || id === "" || id.length > 200) invalid("deleted ids");
return id;
});
if (!document.fullResyncRequired && token === null) invalid("change token");
return { token, changed, deleted, fullResyncRequired: document.fullResyncRequired };
}
/**
* Besides the shape, the file must be a direct child of the directory the app chose,
* after resolving symlinks, so a confused helper cannot point the upload elsewhere.
*/
export async function parseExportOutput(text: string, destination: string): Promise<PhotosExport> {
const document = parseDocument(text);
const { path: file, fileName, sizeBytes, sha256 } = document;
if (
typeof file !== "string" ||
!path.isAbsolute(file) ||
typeof fileName !== "string" ||
fileName === "" ||
typeof sizeBytes !== "number" ||
!Number.isSafeInteger(sizeBytes) ||
sizeBytes < 0 ||
typeof sha256 !== "string" ||
!/^[0-9a-f]{64}$/.test(sha256)
) {
invalid("export result");
}
let real: string;
let realDestination: string;
try {
[real, realDestination] = await Promise.all([realpath(file), realpath(destination)]);
} catch {
invalid("export path");
}
if (path.dirname(real) !== realDestination || path.basename(real) !== fileName) {
throw new PhotosHelperError("path_rejected", "Photos helper wrote outside the export folder.");
}
return { path: real, fileName, sizeBytes, sha256 };
}
/** The last `{"error":{code,message}}` line on stderr; unknown codes become `failed`. */
export function parseHelperError(stderr: string, exitCode: number | null): PhotosHelperError {
const lines = stderr.split("\n").reverse();
for (const line of lines) {
if (!line.startsWith("{")) continue;
try {
const error = (JSON.parse(line) as { error?: unknown }).error as
| { code?: unknown; message?: unknown }
| undefined;
if (typeof error?.code !== "string") continue;
const code = HELPER_CODES.has(error.code as PhotosHelperErrorCode)
? (error.code as PhotosHelperErrorCode)
: "failed";
const message =
typeof error.message === "string" && error.message.length <= 500
? error.message
: "Photos helper failed.";
return new PhotosHelperError(code, message);
} catch {}
}
return new PhotosHelperError("failed", `Photos helper exited with status ${exitCode ?? "?"}.`);
}
/** `{"progress":0.42}` lines on stderr; anything else is ignored. */
export function parseProgressLine(line: string): number | null {
if (!line.startsWith('{"progress"')) return null;
try {
const value = (JSON.parse(line) as { progress?: unknown }).progress;
return typeof value === "number" && value >= 0 && value <= 1 ? value : null;
} catch {
return null;
}
}
// ---------------------------------------------------------------------------
// Process
// ---------------------------------------------------------------------------
/** The subset of ChildProcess the runner needs, so tests can supply a fake. */
export interface HelperProcess {
stdout: Readable | null;
stderr: Readable | null;
kill(signal?: NodeJS.Signals): boolean;
on(event: "error", listener: (error: Error) => void): unknown;
on(event: "close", listener: (code: number | null) => void): unknown;
}
export type SpawnHelper = (
binary: string,
args: string[],
options: { shell: false; env: Record<string, string>; stdio: ["ignore", "pipe", "pipe"] },
) => HelperProcess;
export interface RunOptions {
signal?: AbortSignal;
/** Hard limit for the whole run. */
timeoutMs: number;
/** Fails when stderr progress stops for this long (exports only). */
idleTimeoutMs?: number;
onProgress?: (fraction: number) => void;
}
const MAX_STDOUT_BYTES = 32 * 1024 * 1024;
const MAX_STDERR_BYTES = 1024 * 1024;
const ENV_ALLOWLIST = ["HOME", "USER", "LOGNAME", "TMPDIR", "LANG", "LC_ALL"];
/** PhotoKit needs the user's home; nothing else from the app environment is passed on. */
export function helperEnv(env: NodeJS.ProcessEnv): Record<string, string> {
const result: Record<string, string> = {};
for (const name of ENV_ALLOWLIST) {
const value = env[name];
if (value) result[name] = value;
}
return result;
}
export const defaultSpawnHelper: SpawnHelper = (binary, args, options) =>
nodeSpawn(binary, args, options);
export function runPhotosHelper(
binary: string,
args: string[],
options: RunOptions & { spawn?: SpawnHelper; env?: NodeJS.ProcessEnv },
): Promise<string> {
return new Promise((resolve, reject) => {
if (options.signal?.aborted) {
reject(new PhotosHelperError("cancelled", "Cancelled."));
return;
}
const child = (options.spawn ?? defaultSpawnHelper)(binary, args, {
shell: false,
env: helperEnv(options.env ?? process.env),
stdio: ["ignore", "pipe", "pipe"],
});
const stdout: Buffer[] = [];
let stdoutBytes = 0;
let stderr = "";
let pendingLine = "";
let settled = false;
let idleTimer: NodeJS.Timeout | undefined;
const finish = (error: PhotosHelperError | null, output = "") => {
if (settled) return;
settled = true;
clearTimeout(timer);
clearTimeout(idleTimer);
options.signal?.removeEventListener("abort", abort);
if (error) reject(error);
else resolve(output);
};
const stop = (error: PhotosHelperError) => {
child.kill("SIGTERM");
finish(error);
};
const abort = () => stop(new PhotosHelperError("cancelled", "Cancelled."));
const timer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos helper timed out.")),
options.timeoutMs,
);
timer.unref?.();
const armIdle = () => {
if (options.idleTimeoutMs === undefined) return;
clearTimeout(idleTimer);
idleTimer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos export stalled.")),
options.idleTimeoutMs,
);
idleTimer.unref?.();
};
armIdle();
options.signal?.addEventListener("abort", abort, { once: true });
child.stdout?.on("data", (chunk: Buffer) => {
stdoutBytes += chunk.byteLength;
if (stdoutBytes > MAX_STDOUT_BYTES) {
stop(new PhotosHelperError("invalid_output", "Photos helper output is too large."));
return;
}
stdout.push(chunk);
});
child.stderr?.on("data", (chunk: Buffer) => {
const text = chunk.toString("utf8");
if (stderr.length < MAX_STDERR_BYTES) stderr += text;
const lines = (pendingLine + text).split("\n");
pendingLine = lines.pop() ?? "";
for (const line of lines) {
const progress = parseProgressLine(line.trim());
if (progress !== null) {
armIdle();
options.onProgress?.(progress);
}
}
});
child.on("error", () => {
finish(new PhotosHelperError("unavailable", "Photos helper could not be started."));
});
child.on("close", (code) => {
if (code === 0) finish(null, Buffer.concat(stdout).toString("utf8"));
else finish(parseHelperError(stderr + pendingLine, code));
});
});
}
const QUERY_TIMEOUT_MS = 2 * 60_000;
/** The system permission dialog waits on the person. */
const AUTHORIZE_TIMEOUT_MS = 10 * 60_000;
/** Large iCloud videos take long; a stalled download fails sooner through the idle limit. */
const EXPORT_TIMEOUT_MS = 2 * 60 * 60_000;
const EXPORT_IDLE_TIMEOUT_MS = 5 * 60_000;
export interface PhotosHelper {
authorize(prompt: boolean, signal?: AbortSignal): Promise<PhotosAuthorization>;
list(
request: { cursor: string | null; limit: number; since?: string | null },
signal?: AbortSignal,
): Promise<PhotosListPage>;
asset(localIdentifier: string, signal?: AbortSignal): Promise<PhotosAssetDetail>;
exportOriginal(
localIdentifier: string,
destination: string,
options?: { signal?: AbortSignal; onProgress?: (fraction: number) => void },
): Promise<PhotosExport>;
changes(token: string, signal?: AbortSignal): Promise<PhotosChanges>;
}
export function createPhotosHelper(binary: string, spawn?: SpawnHelper): PhotosHelper {
const run = (args: string[], options: RunOptions) =>
runPhotosHelper(binary, args, { ...options, spawn });
return {
async authorize(prompt, signal) {
const timeoutMs = prompt ? AUTHORIZE_TIMEOUT_MS : QUERY_TIMEOUT_MS;
return parseAuthorizeOutput(await run(authorizeArgs(prompt), { signal, timeoutMs }));
},
async list(request, signal) {
const output = await run(listArgs(request), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseListOutput(output, request.limit);
},
async asset(localIdentifier, signal) {
const output = await run(assetArgs(localIdentifier), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseAssetOutput(output);
},
async exportOriginal(localIdentifier, destination, options = {}) {
const output = await run(exportArgs(localIdentifier, destination), {
signal: options.signal,
onProgress: options.onProgress,
timeoutMs: EXPORT_TIMEOUT_MS,
idleTimeoutMs: EXPORT_IDLE_TIMEOUT_MS,
});
return parseExportOutput(output, destination);
},
async changes(token, signal) {
const output = await run(changesArgs(token), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseChangesOutput(output);
},
};
}

View File

@ -0,0 +1,655 @@
import { createHash } from "node:crypto";
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { ClientMediaItem, MediaAsset, MediaUpload } from "@rakazo/contracts";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { PhotosHelper } from "./media-photos-helper.js";
import type { MediaSyncApi, PhotosSyncState, PhotosSyncStore } from "./media-photos-sync.js";
import {
createChunkUploader,
createPhotosSyncFileStore,
parsePhotosSyncState,
runPool,
syncApplePhotos,
} from "./media-photos-sync.js";
const sha = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
function libraryItem(index: number): ClientMediaItem {
return {
externalId: `ASSET-${String(index).padStart(5, "0")}/L0/001`,
kind: index % 3 === 0 ? "video" : "photo",
capturedAt: new Date(Date.UTC(2024, 0, 1) + index * 60_000).toISOString(),
durationMs: index % 3 === 0 ? 5000 : null,
width: 1920,
height: 1080,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
fileName: `IMG_${index}.JPG`,
};
}
/** Pages a synthetic library the way the Swift helper does: opaque cursor = next index. */
function fakeHelper(options: { size: number; contents?: (id: string) => string }) {
const library = Array.from({ length: options.size }, (_, index) => libraryItem(index));
const exports: string[] = [];
let active = 0;
const stats = { maxActiveExports: 0 };
const helper: PhotosHelper & { library: ClientMediaItem[] } = {
library,
authorize: vi.fn(async () => "authorized" as const),
list: vi.fn(async ({ cursor, limit }) => {
const start = cursor === null ? 0 : Number(cursor);
const items = library.slice(start, start + limit);
const next = start + limit < library.length ? String(start + limit) : null;
return { items, nextCursor: next, changeToken: `token-${library.length}` };
}),
asset: vi.fn(),
changes: vi.fn(),
exportOriginal: vi.fn(async (id: string, destination: string, opts) => {
active += 1;
stats.maxActiveExports = Math.max(stats.maxActiveExports, active);
try {
await new Promise((resolve) => setTimeout(resolve, 5));
opts?.signal?.throwIfAborted();
const contents = options.contents?.(id) ?? `original bytes of ${id}`;
const file = path.join(destination, "IMG.MOV");
await writeFile(file, contents);
exports.push(destination);
return {
path: file,
fileName: "IMG.MOV",
sizeBytes: Buffer.byteLength(contents),
sha256: sha(contents),
};
} finally {
active -= 1;
}
}),
};
return { helper, exports, stats };
}
function memoryStore(initial?: PhotosSyncState) {
let state: PhotosSyncState = initial ?? { version: 1, deviceId: "device-1", targets: {} };
const writes: PhotosSyncState[] = [];
const store: PhotosSyncStore = {
read: async () => JSON.parse(JSON.stringify(state)) as PhotosSyncState,
write: async (next) => {
state = JSON.parse(JSON.stringify(next)) as PhotosSyncState;
writes.push(state);
},
};
return { store, writes, current: () => state };
}
function asset(id: string, externalId: string, ingestState: MediaAsset["ingestState"]) {
return {
id,
sourceId: "source-1",
externalId,
kind: "video",
storage: "remote",
capturedAt: null,
durationMs: null,
width: null,
height: null,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
ingestState,
hasProxy: false,
hasThumbnail: false,
qualityScore: null,
createdAt: "2024-01-01T00:00:00.000Z",
} as MediaAsset;
}
function fakeApi(
options: {
assets?: MediaAsset[];
chunkBytes?: number;
uploads?: Record<string, MediaUpload>;
} = {},
) {
const uploads = new Map(Object.entries(options.uploads ?? {}));
const pushes: {
items: number;
syncToken: string | null;
complete: boolean;
deleted?: number;
}[] = [];
const chunkBytes = options.chunkBytes ?? 4;
const api = {
media: {
sources: {
registerClient: vi.fn(async (input: { deviceId: string }) => ({
id: "source-1",
kind: "apple_photos" as const,
label: "Apple Photos",
status: "idle" as const,
assetCount: 0,
lastSyncedAt: null,
error: null,
createdAt: "2024-01-01T00:00:00.000Z",
deviceId: input.deviceId,
})),
pushItems: vi.fn(
async (input: {
items: unknown[];
deletedExternalIds?: string[];
syncToken: string | null;
complete: boolean;
}) => {
const deleted = input.deletedExternalIds?.length ?? 0;
expect(input.items.length > 0 || deleted > 0 || input.complete).toBe(true);
expect(input.items.length).toBeLessThanOrEqual(500);
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
...(deleted > 0 ? { deleted } : {}),
});
return { created: input.items.length, updated: 0, unchanged: 0, deleted };
},
),
},
assets: {
search: vi.fn(async (input: { cursor?: string | null }) => {
const all = options.assets ?? [];
// Two pages to exercise pagination.
const half = Math.ceil(all.length / 2);
return input.cursor === "p2"
? { items: all.slice(half), nextCursor: null }
: { items: all.slice(0, half), nextCursor: all.length > 1 ? "p2" : null };
}),
},
uploads: {
begin: vi.fn(async (input: { assetId: string; sizeBytes: number }) => {
const upload: MediaUpload = {
id: `upload-${input.assetId}`,
assetId: input.assetId,
sizeBytes: input.sizeBytes,
chunkBytes,
receivedChunks: [],
status: "open",
};
uploads.set(upload.id, upload);
return upload;
}),
status: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId);
if (!upload) throw Object.assign(new Error("Not found"), { code: "NOT_FOUND" });
return upload;
}),
complete: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId)!;
return { ...upload, status: "complete" as const };
}),
},
},
};
return { api: api as unknown as MediaSyncApi & typeof api, pushes };
}
let tempRoot: string;
beforeEach(async () => {
tempRoot = await mkdtemp(path.join(tmpdir(), "rakazo-photos-sync-"));
});
afterEach(async () => {
await rm(tempRoot, { recursive: true, force: true });
});
const baseOptions = () => ({
target: "https://rakazo.example.com#space_1",
label: "Apple Photos",
tempRoot,
signal: new AbortController().signal,
});
describe("Apple Photos metadata sync", () => {
it("registers once, pushes batches of at most 500 and completes the listing", async () => {
const { helper } = fakeHelper({ size: 1203 });
const { api, pushes } = fakeApi();
const { store, current } = memoryStore();
const chunks = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledWith(
{ kind: "apple_photos", label: "Apple Photos", deviceId: "device-1" },
expect.anything(),
);
expect(pushes).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 500, syncToken: null, complete: false },
{ items: 203, syncToken: "token-1203", complete: true },
]);
expect(result).toMatchObject({ indexed: 1203, created: 1203, uploaded: 0, fullListing: true });
expect(current().targets["https://rakazo.example.com#space_1"]).toEqual({
sourceId: "source-1",
listing: null,
changeToken: "token-1203",
uploads: {},
});
// The next run reads changes only and does not register again.
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "token-next",
changed: [libraryItem(1), libraryItem(2)],
deleted: [],
fullResyncRequired: false,
});
const second = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(helper.changes).toHaveBeenCalledWith("token-1203", expect.anything());
expect(pushes.at(-1)).toEqual({ items: 2, syncToken: "token-next", complete: false });
expect(second.fullListing).toBe(false);
expect(current().targets["https://rakazo.example.com#space_1"]?.changeToken).toBe("token-next");
});
it("persists the cursor after every batch and resumes an interrupted listing", async () => {
const { helper } = fakeHelper({ size: 1100 });
const { api, pushes } = fakeApi();
const { store, writes, current } = memoryStore();
vi.mocked(api.media.sources.pushItems).mockImplementationOnce(async (input) => {
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
});
return { created: input.items.length, updated: 0, unchanged: 0 };
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(new Error("network down"));
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() }),
).rejects.toThrow("network down");
const interrupted = current().targets["https://rakazo.example.com#space_1"]!;
expect(interrupted.listing).toMatchObject({ cursor: "500", token: "token-1100" });
expect(interrupted.listing?.lastItem?.externalId).toBe(libraryItem(499).externalId);
expect(writes.length).toBeGreaterThan(1);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(vi.mocked(helper.list).mock.calls.at(-2)?.[0]).toMatchObject({ cursor: "500" });
expect(pushes.slice(1)).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 100, syncToken: "token-1100", complete: true },
]);
expect(current().targets["https://rakazo.example.com#space_1"]).toMatchObject({
listing: null,
changeToken: "token-1100",
});
});
it("closes a resumed listing whose last page came back empty", async () => {
const { helper } = fakeHelper({ size: 0 });
const { api, pushes } = fakeApi();
const last = libraryItem(7);
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: { cursor: "500", token: "token-a", lastItem: last },
changeToken: null,
uploads: {},
},
},
});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(pushes).toEqual([{ items: 0, syncToken: "token-a", complete: true }]);
});
it("pushes device deletions without relisting", async () => {
const { helper } = fakeHelper({ size: 3 });
const { api, pushes } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "old",
uploads: {},
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "new",
changed: [],
deleted: ["gone/L0/001"],
fullResyncRequired: false,
});
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: vi.fn(),
});
expect(result.fullListing).toBe(false);
expect(pushes).toEqual([{ items: 0, syncToken: "new", complete: false, deleted: 1 }]);
});
it("registers again when the server no longer knows the source", async () => {
const { helper } = fakeHelper({ size: 2 });
const { api } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "deleted-source",
listing: { cursor: null, token: null, lastItem: null },
changeToken: null,
uploads: {},
},
},
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(
Object.assign(new Error("Not found"), { code: "NOT_FOUND" }),
);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(vi.mocked(api.media.sources.pushItems).mock.calls.at(-1)?.[0].sourceId).toBe("source-1");
});
it("stops before anything is sent when Photos access is denied", async () => {
const { helper } = fakeHelper({ size: 2 });
vi.mocked(helper.authorize).mockResolvedValueOnce("denied");
const { api } = fakeApi();
await expect(
syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk: vi.fn(),
}),
).rejects.toMatchObject({ code: "not_authorized" });
expect(api.media.sources.registerClient).not.toHaveBeenCalled();
});
});
describe("Apple Photos original upload", () => {
const awaitingAssets = [
asset("a1", "ASSET-00001/L0/001", "awaiting_upload"),
asset("a2", "ASSET-00002/L0/001", "proxied"),
asset("a3", "ASSET-00003/L0/001", "awaiting_upload"),
asset("a4", "ASSET-00004/L0/001", "awaiting_upload"),
];
it("uploads every awaiting original in chunks with two at a time and cleans up", async () => {
const { helper, exports, stats } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: awaitingAssets, chunkBytes: 8 });
const sent = new Map<string, Buffer[]>();
const uploadChunk = vi.fn(async (uploadId: string, index: number, bytes: Uint8Array) => {
const parts = sent.get(uploadId) ?? [];
parts[index] = Buffer.from(bytes);
sent.set(uploadId, parts);
});
const progress = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk,
onProgress: progress,
});
expect(result.uploaded).toBe(3);
expect(
vi
.mocked(helper.exportOriginal)
.mock.calls.map((call) => call[0])
.sort(),
).toEqual(["ASSET-00001/L0/001", "ASSET-00003/L0/001", "ASSET-00004/L0/001"]);
expect(stats.maxActiveExports).toBe(2);
expect(Buffer.concat(sent.get("upload-a1")!).toString()).toBe(
"original bytes of ASSET-00001/L0/001",
);
expect(sent.get("upload-a1")).toHaveLength(Math.ceil(36 / 8));
expect(api.media.uploads.begin).toHaveBeenCalledWith(
expect.objectContaining({
assetId: "a1",
sizeBytes: 36,
sha256: sha("original bytes of ASSET-00001/L0/001"),
}),
expect.anything(),
);
expect(api.media.uploads.complete).toHaveBeenCalledTimes(3);
expect(progress).toHaveBeenLastCalledWith({
phase: "uploading",
indexed: 1,
uploadsDone: 3,
uploadsTotal: 3,
});
for (const dir of exports) await expect(readdir(dir)).rejects.toThrow();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("resumes an open upload and sends only the missing chunks", async () => {
const { helper } = fakeHelper({ size: 1 });
const contents = "original bytes of ASSET-00001/L0/001";
const { api } = fakeApi({
assets: [awaitingAssets[0]!],
uploads: {
"upload-old": {
id: "upload-old",
assetId: "a1",
sizeBytes: Buffer.byteLength(contents),
chunkBytes: 10,
receivedChunks: [0, 2],
status: "open",
},
},
});
const { store, current } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "tok",
uploads: { a1: "upload-old" },
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "tok",
changed: [],
deleted: [],
fullResyncRequired: false,
});
const uploadChunk = vi.fn(async () => {});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(
uploadChunk.mock.calls.map((call) => [
call[0],
call[1],
Buffer.from(call[2] as Uint8Array).toString(),
]),
).toEqual([
["upload-old", 1, contents.slice(10, 20)],
["upload-old", 3, contents.slice(30)],
]);
expect(current().targets["https://rakazo.example.com#space_1"]?.uploads).toEqual({});
});
it("aborts on a checksum mismatch without starting an upload", async () => {
const { helper } = fakeHelper({ size: 1 });
vi.mocked(helper.exportOriginal).mockImplementationOnce(async (_id, destination) => {
const file = path.join(destination, "IMG.MOV");
await writeFile(file, "tampered");
return { path: file, fileName: "IMG.MOV", sizeBytes: 8, sha256: sha("original") };
});
const { api } = fakeApi({ assets: [awaitingAssets[0]!] });
const uploadChunk = vi.fn();
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store: memoryStore().store, uploadChunk }),
).rejects.toMatchObject({ code: "integrity" });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(uploadChunk).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("cancels a running upload", async () => {
const { helper } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: [awaitingAssets[0]!], chunkBytes: 4 });
const controller = new AbortController();
const uploadChunk = vi.fn(
async (_id: string, index: number, _bytes: Uint8Array, signal: AbortSignal) => {
if (index === 1) controller.abort();
signal.throwIfAborted();
},
);
await expect(
syncApplePhotos({
...baseOptions(),
signal: controller.signal,
helper,
api,
store: memoryStore().store,
uploadChunk,
}),
).rejects.toMatchObject({ code: "cancelled" });
expect(uploadChunk).toHaveBeenCalledTimes(2);
expect(api.media.uploads.complete).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
});
describe("Apple Photos sync state file", () => {
it("keeps a stable device id across launches", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-state-"));
try {
const store = createPhotosSyncFileStore(dir, () => "device-a");
const first = await store.read();
expect(first.deviceId).toBe("device-a");
first.targets.x = { sourceId: "s", listing: null, changeToken: "t", uploads: {} };
await store.write(first);
const reopened = await createPhotosSyncFileStore(dir, () => "device-b").read();
expect(reopened).toEqual(first);
expect(await readFile(path.join(dir, "media-photos-sync.json"), "utf8")).toContain(
"device-a",
);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("drops unreadable state and malformed targets", () => {
expect(parsePhotosSyncState("{")).toBeNull();
expect(
parsePhotosSyncState(JSON.stringify({ version: 2, deviceId: "d", targets: {} })),
).toBeNull();
expect(
parsePhotosSyncState(
JSON.stringify({
version: 1,
deviceId: "d",
targets: {
good: { sourceId: "s", listing: null, changeToken: null, uploads: {} },
bad: { sourceId: 4, listing: null, changeToken: null, uploads: {} },
badListing: { sourceId: "s", listing: { cursor: 3 }, changeToken: null, uploads: {} },
},
}),
)?.targets,
).toEqual({ good: { sourceId: "s", listing: null, changeToken: null, uploads: {} } });
});
});
describe("chunk uploader", () => {
it("PUTs raw bytes with the session and space", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 204 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: "space_1",
fetch,
});
const signal = new AbortController().signal;
await upload("up/1", 3, new Uint8Array([1, 2, 3]), signal);
expect(fetch).toHaveBeenCalledWith(
"https://rakazo.example.com/api/media/uploads/up%2F1/chunks/3",
expect.objectContaining({
method: "PUT",
credentials: "include",
redirect: "error",
signal,
headers: { "content-type": "application/octet-stream", "x-rakazo-space-id": "space_1" },
}),
);
});
it("rejects failed chunks and non-origin targets", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 409 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: null,
fetch,
});
await expect(
upload("u", 0, new Uint8Array(), new AbortController().signal),
).rejects.toMatchObject({
code: "upload_failed",
});
expect(() =>
createChunkUploader({ origin: "https://rakazo.example.com/rpc", spaceId: null, fetch }),
).toThrow();
expect(() => createChunkUploader({ origin: "file:///etc", spaceId: null, fetch })).toThrow();
});
});
describe("runPool", () => {
it("bounds concurrency and reports the first failure", async () => {
let active = 0;
let max = 0;
const seen: number[] = [];
await expect(
runPool([1, 2, 3, 4, 5], 2, new AbortController().signal, async (item, signal) => {
active += 1;
max = Math.max(max, active);
await new Promise((resolve) => setTimeout(resolve, 5));
active -= 1;
if (item === 2) throw new Error("boom");
signal.throwIfAborted();
seen.push(item);
}),
).rejects.toThrow("boom");
expect(max).toBe(2);
expect(seen).not.toContain(4);
});
});

View File

@ -0,0 +1,581 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { mkdtemp, open, rm, stat } from "node:fs/promises";
import path from "node:path";
import type {
ClientMediaItem,
MediaAsset,
MediaUpload,
mediaRpcShapes,
PhotosSyncProgress,
PhotosSyncResult,
} from "@rakazo/contracts";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PHOTOS_MAX_PAGE } from "./media-photos-helper.js";
import { readPrivateFile, writePrivateFile } from "./setup-store.js";
/*
* Apple Photos is a client-pushed source: the API may run on another machine, so the
* app never sends a local path. It pushes metadata in batches and uploads the originals
* the server asks for (`awaiting_upload`) in resumable raw-byte chunks.
*/
type Shapes = typeof mediaRpcShapes;
type Shape = { input: { _input: unknown }; output: { _output: unknown } };
/** One oRPC procedure as `ContractRouterClient<AppContract>` exposes it. */
type Procedure<S extends Shape> = (
input: S["input"]["_input"],
options?: { signal?: AbortSignal },
) => Promise<S["output"]["_output"]>;
/** The slice of the app RPC client the sync needs; `rpc` from a contract client satisfies it. */
export interface MediaSyncApi {
media: {
sources: {
registerClient: Procedure<Shapes["sources"]["registerClient"]>;
pushItems: Procedure<Shapes["sources"]["pushItems"]>;
};
assets: { search: Procedure<Shapes["assets"]["search"]> };
uploads: {
begin: Procedure<Shapes["uploads"]["begin"]>;
status: Procedure<Shapes["uploads"]["status"]>;
complete: Procedure<Shapes["uploads"]["complete"]>;
};
};
}
/** Sends one raw chunk to `PUT /api/media/uploads/:uploadId/chunks/:index`. */
export type UploadChunk = (
uploadId: string,
index: number,
bytes: Uint8Array<ArrayBuffer>,
signal: AbortSignal,
) => Promise<void>;
export type PhotosSyncErrorCode = "not_authorized" | "cancelled" | "integrity" | "upload_failed";
export class PhotosSyncError extends Error {
constructor(
readonly code: PhotosSyncErrorCode,
message: string,
) {
super(message);
this.name = "PhotosSyncError";
}
}
// ---------------------------------------------------------------------------
// Persistent state
// ---------------------------------------------------------------------------
export interface PhotosSyncTargetState {
sourceId: string;
/** A full listing in progress: where to resume, the token from its start and the last pushed item. */
listing: { cursor: string | null; token: string | null; lastItem: ClientMediaItem | null } | null;
/** Set after a complete listing; later syncs only read changes since it. */
changeToken: string | null;
/** Open uploads by asset id so an interrupted upload resumes instead of restarting. */
uploads: Record<string, string>;
}
export interface PhotosSyncState {
version: 1;
/** Stable per installation; the server keys the Apple Photos source by it. */
deviceId: string;
/** Keyed by server origin and space, since each has its own source. */
targets: Record<string, PhotosSyncTargetState>;
}
export interface PhotosSyncStore {
read(): Promise<PhotosSyncState>;
write(state: PhotosSyncState): Promise<void>;
}
export const PHOTOS_SYNC_FILE_NAME = "media-photos-sync.json";
const MAX_STATE_BYTES = 4 * 1024 * 1024;
/** Kept next to the desktop setup file, private to the user and written atomically. */
export function createPhotosSyncFileStore(
userDataDir: string,
newDeviceId: () => string = randomUUID,
): PhotosSyncStore {
const file = path.join(userDataDir, PHOTOS_SYNC_FILE_NAME);
return {
async read() {
const raw = await readPrivateFile(file, MAX_STATE_BYTES);
return (raw === null ? null : parsePhotosSyncState(raw)) ?? emptyState(newDeviceId());
},
write: (state) => writePrivateFile(file, `${JSON.stringify(state)}\n`),
};
}
function emptyState(deviceId: string): PhotosSyncState {
return { version: 1, deviceId, targets: {} };
}
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" && value !== null && !Array.isArray(value);
const isNullableString = (value: unknown) => value === null || typeof value === "string";
/** Unreadable state starts over: the server deduplicates by device and external id. */
export function parsePhotosSyncState(raw: string): PhotosSyncState | null {
let value: unknown;
try {
value = JSON.parse(raw);
} catch {
return null;
}
if (
!isRecord(value) ||
value.version !== 1 ||
typeof value.deviceId !== "string" ||
value.deviceId === "" ||
value.deviceId.length > 200 ||
!isRecord(value.targets)
) {
return null;
}
const targets: Record<string, PhotosSyncTargetState> = {};
for (const [key, target] of Object.entries(value.targets)) {
if (
!isRecord(target) ||
typeof target.sourceId !== "string" ||
!isNullableString(target.changeToken) ||
!isRecord(target.uploads) ||
!Object.values(target.uploads).every((id) => typeof id === "string")
) {
continue;
}
const listing = target.listing;
const validListing =
listing === null ||
(isRecord(listing) &&
isNullableString(listing.cursor) &&
isNullableString(listing.token) &&
(listing.lastItem === null || isRecord(listing.lastItem)));
if (!validListing) continue;
targets[key] = target as unknown as PhotosSyncTargetState;
}
return { version: 1, deviceId: value.deviceId, targets };
}
// ---------------------------------------------------------------------------
// Chunk upload over HTTP
// ---------------------------------------------------------------------------
/**
* Chunk PUTs ride the same signed-in session as the app window: the injected `fetch`
* must carry that session's cookies (Electron `session.fetch` with credentials).
*/
export function createChunkUploader(options: {
origin: string;
spaceId: string | null;
fetch: (url: string, init: RequestInit) => Promise<Response>;
}): UploadChunk {
const origin = new URL(options.origin);
if (!["http:", "https:"].includes(origin.protocol) || origin.origin !== options.origin) {
throw new Error("Uploads need the server origin.");
}
return async (uploadId, index, bytes, signal) => {
if (!Number.isSafeInteger(index) || index < 0) throw new Error("Invalid chunk index.");
const headers: Record<string, string> = { "content-type": "application/octet-stream" };
if (options.spaceId) headers["x-rakazo-space-id"] = options.spaceId;
const response = await options.fetch(
`${origin.origin}/api/media/uploads/${encodeURIComponent(uploadId)}/chunks/${index}`,
{
method: "PUT",
headers,
body: bytes,
credentials: "include",
redirect: "error",
signal,
},
);
if (!response.ok) {
throw new PhotosSyncError(
"upload_failed",
`The server rejected an upload (${response.status}).`,
);
}
};
}
// ---------------------------------------------------------------------------
// Sync
// ---------------------------------------------------------------------------
export type { PhotosSyncProgress, PhotosSyncResult };
export interface PhotosSyncOptions {
helper: PhotosHelper;
api: MediaSyncApi;
uploadChunk: UploadChunk;
store: PhotosSyncStore;
/** Server origin plus space, e.g. `https://rakazo.example.com#space_1`. */
target: string;
/** Source label shown in the library, e.g. the Mac's name or "Apple Photos". */
label: string;
/** Exports land in a fresh folder below this and are deleted after upload. */
tempRoot: string;
signal: AbortSignal;
onProgress?: (progress: PhotosSyncProgress) => void;
/** At most 500, the contract's batch limit. */
batchSize?: number;
concurrency?: number;
}
const MAX_SYNC_TOKEN_LENGTH = 2000;
export async function syncApplePhotos(options: PhotosSyncOptions): Promise<PhotosSyncResult> {
const { helper, api, signal } = options;
const batchSize = Math.min(Math.max(1, options.batchSize ?? PHOTOS_MAX_PAGE), PHOTOS_MAX_PAGE);
const result: PhotosSyncResult = {
indexed: 0,
created: 0,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: false,
};
const progress = { uploadsDone: 0, uploadsTotal: 0 };
const report = (phase: PhotosSyncProgress["phase"]) =>
options.onProgress?.({ phase, indexed: result.indexed, ...progress });
const state = await options.store.read();
// Writes are serialized so concurrent upload workers never interleave partial state.
let saving = Promise.resolve();
const save = () => {
const snapshot = JSON.parse(JSON.stringify(state)) as PhotosSyncState;
saving = saving.then(() => options.store.write(snapshot));
return saving;
};
try {
report("authorizing");
const authorization = await helper.authorize(true, signal);
if (authorization !== "authorized" && authorization !== "limited") {
throw new PhotosSyncError("not_authorized", "Photos access was not granted.");
}
const ensureTarget = async (): Promise<PhotosSyncTargetState> => {
const existing = state.targets[options.target];
if (existing) return existing;
const source = await api.media.sources.registerClient(
{ kind: "apple_photos", label: options.label, deviceId: state.deviceId },
{ signal },
);
const created: PhotosSyncTargetState = {
sourceId: source.id,
listing: null,
changeToken: null,
uploads: {},
};
state.targets[options.target] = created;
await save();
return created;
};
const push = async (
target: PhotosSyncTargetState,
items: ClientMediaItem[],
syncToken: string | null,
complete: boolean,
deletedExternalIds: string[] = [],
) => {
const counts = await api.media.sources.pushItems(
{
sourceId: target.sourceId,
items,
deletedExternalIds,
syncToken:
syncToken !== null && syncToken.length <= MAX_SYNC_TOKEN_LENGTH ? syncToken : null,
complete,
},
{ signal },
);
result.indexed += items.length;
result.created += counts.created;
result.updated += counts.updated;
result.unchanged += counts.unchanged;
report("indexing");
};
/** Full listing, resumable page by page; the last push says `complete`. */
const listAll = async (target: PhotosSyncTargetState) => {
result.fullListing = true;
target.listing ??= { cursor: null, token: null, lastItem: null };
const listing = target.listing;
while (true) {
const page = await helper.list({ cursor: listing.cursor, limit: batchSize }, signal);
if (listing.cursor === null) listing.token = page.changeToken;
const done = page.nextCursor === null;
const items = page.items;
// An empty final page still closes the listing.
if (items.length > 0 || done) await push(target, items, done ? listing.token : null, done);
if (done) {
target.changeToken = listing.token;
target.listing = null;
await save();
return;
}
listing.cursor = page.nextCursor;
listing.lastItem = items.at(-1) ?? listing.lastItem;
await save();
}
};
const syncMetadata = async () => {
const target = await ensureTarget();
if (target.listing !== null || target.changeToken === null) {
await listAll(target);
return;
}
const changes = await helper.changes(target.changeToken, signal);
if (changes.fullResyncRequired || changes.token === null) {
target.listing = null;
await listAll(target);
return;
}
const batches = Math.max(
Math.ceil(changes.changed.length / batchSize),
Math.ceil(changes.deleted.length / DELETED_BATCH_SIZE),
);
for (let index = 0; index < batches; index++) {
const batch = changes.changed.slice(index * batchSize, (index + 1) * batchSize);
const deleted = changes.deleted.slice(
index * DELETED_BATCH_SIZE,
(index + 1) * DELETED_BATCH_SIZE,
);
const last = index === batches - 1;
await push(target, batch, last ? changes.token : null, false, deleted);
}
target.changeToken = changes.token;
await save();
};
report("indexing");
try {
await syncMetadata();
} catch (error) {
// The source was removed on the server: register again and list from scratch.
if (!isNotFound(error) || signal.aborted) throw error;
delete state.targets[options.target];
await save();
await syncMetadata();
}
const target = state.targets[options.target]!;
const awaiting = await awaitingUploads(api, target.sourceId, signal);
progress.uploadsTotal = awaiting.length;
// Forget uploads for assets the server no longer waits for.
const waiting = new Set(awaiting.map((asset) => asset.id));
for (const assetId of Object.keys(target.uploads)) {
if (!waiting.has(assetId)) delete target.uploads[assetId];
}
await save();
report("uploading");
await runPool(awaiting, options.concurrency ?? 2, signal, async (asset, workerSignal) => {
await uploadAsset({ ...options, signal: workerSignal }, target, asset, save);
result.uploaded += 1;
progress.uploadsDone += 1;
report("uploading");
});
return result;
} catch (error) {
if (signal.aborted) throw new PhotosSyncError("cancelled", "Sync cancelled.");
throw error;
} finally {
await saving.catch(() => undefined);
}
}
/**
* Binds the sync to the server and space the request names. `connect` builds the RPC
* client and chunk uploader on the signed-in session for that space.
*/
export function createPhotosSyncRunner(deps: {
helper: PhotosHelper;
store: PhotosSyncStore;
tempRoot: string;
label: string;
connect: (
spaceId: string | null,
) => Promise<{ origin: string; api: MediaSyncApi; uploadChunk: UploadChunk }>;
}) {
return async (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}): Promise<PhotosSyncResult> => {
const { origin, api, uploadChunk } = await deps.connect(request.spaceId);
return syncApplePhotos({
helper: deps.helper,
store: deps.store,
tempRoot: deps.tempRoot,
label: deps.label,
api,
uploadChunk,
target: `${origin}#${request.spaceId ?? "default"}`,
signal: request.signal,
onProgress: request.onProgress,
});
};
}
function isNotFound(error: unknown): boolean {
return isRecord(error) && error.code === "NOT_FOUND";
}
interface AwaitingAsset {
id: string;
externalId: string;
}
/** Assets of this source the server is waiting for, mapped back to Photos items. */
const DELETED_BATCH_SIZE = 5000;
export async function awaitingUploads(
api: MediaSyncApi,
sourceId: string,
signal: AbortSignal,
): Promise<AwaitingAsset[]> {
const awaiting: AwaitingAsset[] = [];
let cursor: string | null = null;
do {
const page: { items: MediaAsset[]; nextCursor: string | null } = await api.media.assets.search(
{ sourceId, ingestStates: ["awaiting_upload"], cursor, limit: PHOTOS_MAX_PAGE },
{ signal },
);
for (const asset of page.items) {
if (asset.ingestState !== "awaiting_upload" || asset.sourceId !== sourceId) continue;
awaiting.push({ id: asset.id, externalId: asset.externalId });
}
cursor = page.nextCursor;
} while (cursor !== null);
return awaiting;
}
async function uploadAsset(
options: PhotosSyncOptions,
target: PhotosSyncTargetState,
asset: AwaitingAsset,
save: () => Promise<void>,
): Promise<void> {
const { api, signal } = options;
const directory = await mkdtemp(path.join(options.tempRoot, "rakazo-photos-"));
try {
const exported = await options.helper.exportOriginal(asset.externalId, directory, { signal });
// The helper's hash is checked against the bytes on disk before anything is sent.
const local = await hashFile(exported.path, signal);
if (local.sha256 !== exported.sha256 || local.sizeBytes !== exported.sizeBytes) {
throw new PhotosSyncError("integrity", "The exported file does not match its checksum.");
}
let upload: MediaUpload | null = null;
const previous = target.uploads[asset.id];
if (previous) {
const status = await api.media.uploads
.status({ uploadId: previous }, { signal })
.catch((error: unknown) => {
if (isNotFound(error)) return null;
throw error;
});
if (status?.status === "complete") {
delete target.uploads[asset.id];
await save();
return;
}
if (status?.status === "open" && status.sizeBytes === local.sizeBytes) upload = status;
}
if (upload === null) {
upload = await api.media.uploads.begin(
{
assetId: asset.id,
sizeBytes: local.sizeBytes,
sha256: local.sha256,
fileName: exported.fileName.slice(0, 255),
},
{ signal },
);
target.uploads[asset.id] = upload.id;
await save();
}
const received = new Set(upload.receivedChunks);
const chunkCount = Math.ceil(local.sizeBytes / upload.chunkBytes);
const file = await open(exported.path, "r");
try {
for (let index = 0; index < chunkCount; index += 1) {
if (received.has(index)) continue;
signal.throwIfAborted();
const offset = index * upload.chunkBytes;
const length = Math.min(upload.chunkBytes, local.sizeBytes - offset);
const bytes = Buffer.alloc(length);
const { bytesRead } = await file.read(bytes, 0, length, offset);
if (bytesRead !== length) {
throw new PhotosSyncError("integrity", "The exported file changed during upload.");
}
await options.uploadChunk(upload.id, index, bytes, signal);
}
} finally {
await file.close();
}
const completed = await api.media.uploads.complete({ uploadId: upload.id }, { signal });
if (completed.status !== "complete") {
throw new PhotosSyncError("upload_failed", "The server did not accept the upload.");
}
delete target.uploads[asset.id];
await save();
} finally {
await rm(directory, { recursive: true, force: true });
}
}
export async function hashFile(
file: string,
signal?: AbortSignal,
): Promise<{ sha256: string; sizeBytes: number }> {
const hash = createHash("sha256");
for await (const chunk of createReadStream(file, { signal })) hash.update(chunk as Buffer);
return { sha256: hash.digest("hex"), sizeBytes: (await stat(file)).size };
}
/**
* Runs `work` with at most `limit` items in flight. The first failure aborts the
* others and is the error reported.
*/
export async function runPool<T>(
items: T[],
limit: number,
signal: AbortSignal,
work: (item: T, signal: AbortSignal) => Promise<void>,
): Promise<void> {
const controller = new AbortController();
const forward = () => controller.abort(signal.reason);
signal.addEventListener("abort", forward, { once: true });
if (signal.aborted) forward();
let next = 0;
let failed = false;
let failure: unknown;
const worker = async () => {
while (!failed && !controller.signal.aborted && next < items.length) {
const item = items[next++]!;
try {
await work(item, controller.signal);
} catch (error) {
if (!failed) {
failed = true;
failure = error;
controller.abort(error);
}
}
}
};
try {
await Promise.all(Array.from({ length: Math.min(Math.max(1, limit), items.length) }, worker));
} finally {
signal.removeEventListener("abort", forward);
}
if (failed) throw failure;
signal.throwIfAborted();
}

View File

@ -9,6 +9,18 @@ contextBridge.exposeInMainWorld("rakazoDesktop", {
request: (pathname, body) =>
ipcRenderer.invoke("desktop.localSettings.request", pathname, body),
},
media: {
photos: {
status: () => ipcRenderer.invoke("desktop.media.photos.status"),
sync: (request) => ipcRenderer.invoke("desktop.media.photos.sync", request),
cancel: () => ipcRenderer.invoke("desktop.media.photos.cancel"),
onProgress: (listener) => {
const handler = (_event, progress) => listener(progress);
ipcRenderer.on("desktop.media.photos.progress", handler);
return () => ipcRenderer.off("desktop.media.photos.progress", handler);
},
},
},
window: {
close: () => ipcRenderer.invoke("desktop.window.close"),
minimize: () => ipcRenderer.invoke("desktop.window.minimize"),

View File

@ -35,6 +35,7 @@ describe("desktop preload bridge", () => {
"browserPreview",
"files",
"localSettings",
"media",
"oauth",
"platform",
"update",
@ -47,6 +48,12 @@ describe("desktop preload bridge", () => {
"toggleMaximize",
]);
expect(Object.keys(bridge.update).sort()).toEqual(["check", "download", "install", "state"]);
expect(Object.keys(bridge.media?.photos ?? {}).sort()).toEqual([
"cancel",
"onProgress",
"status",
"sync",
]);
expect(Object.keys(bridge.browserPreview ?? {}).sort()).toEqual([
"back",
"forward",
@ -97,6 +104,7 @@ describe("desktop preload bridge", () => {
"browserPreview",
"files",
"localSettings",
"media",
"oauth",
"platform",
"update",

View File

@ -35,6 +35,7 @@
"expo-image-picker": "57.0.20",
"expo-linking": "57.0.11",
"expo-localization": "57.0.2",
"expo-media-library": "57.0.5",
"expo-modules-core": "57.0.20",
"expo-notifications": "57.0.21",
"expo-router": "57.0.24",
@ -44,6 +45,7 @@
"expo-status-bar": "57.0.1",
"expo-symbols": "57.0.3",
"expo-updates": "57.0.24",
"expo-video": "57.0.5",
"react": "19.2.3",
"react-dom": "19.2.3",
"react-native": "0.86.3",

View File

@ -0,0 +1,452 @@
import { expect, type Page, type Route, test } from "@playwright/test";
import { captureScreenshot, completeOnboarding, rpc, signup } from "./helpers";
/*
* Media Studio library, collection, people and results views on the scripted
* runtime (media emulators enabled). The library is seeded through the device
* source RPCs with synthetic metadata; people and results need analysis and
* publications the scripted stack does not produce, so those RPCs are stubbed
* with synthetic records. Every name and value below is fake.
*/
type Json = Record<string, unknown>;
const DAY_MS = 24 * 60 * 60 * 1000;
function syntheticItems(count: number) {
const start = Date.UTC(2026, 4, 2, 9);
const cities = [
{ lat: 52.52, lon: 13.4 },
{ lat: 48.14, lon: 11.58 },
{ lat: 53.55, lon: 9.99 },
];
return Array.from({ length: count }, (_, index) => {
const session = index % 3;
const city = cities[session] ?? cities[0];
const photo = index % 5 === 4;
return {
externalId: `e2e-clip-${index}`,
kind: photo ? "photo" : "video",
capturedAt: new Date(start + session * 3 * DAY_MS + index * 60_000).toISOString(),
durationMs: photo ? null : 4000 + (index % 4) * 1000,
width: 1920,
height: 1080,
fps: photo ? null : 30,
gpsLat: city?.lat ?? null,
gpsLon: city?.lon ?? null,
deviceModel: null,
albumNames: [],
isFavorite: index % 7 === 0,
fileName: `clip-${index}.${photo ? "jpg" : "mov"}`,
};
});
}
async function fulfill(route: Route, json: unknown) {
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ json }),
});
}
async function stub(page: Page, procedure: string, respond: (input: Json) => unknown) {
await page.route(`**/rpc/${procedure}`, async (route) => {
const body = (route.request().postDataJSON() ?? {}) as { json?: Json };
await fulfill(route, respond(body.json ?? {}));
});
}
const now = "2026-10-01T12:00:00.000Z";
function person(id: string, name: string, overrides: Json = {}) {
return {
id,
displayName: name,
isSelf: false,
confirmed: false,
anonymizeByDefault: "none",
consentRecordedAt: null,
consentArtifactId: null,
ageVerifiedAt: null,
ageVerificationMethod: "none",
faceTrackCount: 3,
coverThumbAssetId: null,
...overrides,
};
}
function syntheticAsset(index: number) {
return {
id: `scale-${index}`,
sourceId: "scale-source",
externalId: `scale-${index}`,
kind: "video",
storage: "remote",
capturedAt: now,
durationMs: 4000,
width: 1920,
height: 1080,
fps: 30,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
ingestState: "indexed",
hasProxy: false,
hasThumbnail: false,
qualityScore: null,
createdAt: now,
};
}
async function openMediaStudio(page: Page) {
await expect(page.getByTestId("media-studio-open")).toBeVisible({ timeout: 30_000 });
await page.getByTestId("media-studio-open").click();
const studio = page.getByTestId("media-studio");
await expect(studio).toBeVisible();
return studio;
}
test("media library groups sessions, selects clips and creates a project", async ({
page,
}, testInfo) => {
test.setTimeout(240_000);
const stamp = Date.now();
await signup(page, `media-library-${stamp}@rakazo.test`, "password12", "Media Library");
await completeOnboarding(page);
const capabilities = await rpc<{ configured: boolean }>(page, "media/capabilities", {});
expect(capabilities.configured).toBe(true);
const source = await rpc<{ id: string }>(page, "media/sources/registerClient", {
kind: "ios_library",
label: "Test phone",
deviceId: `e2e-device-${stamp}`,
});
const pushed = await rpc<{ created: number }>(page, "media/sources/pushItems", {
sourceId: source.id,
items: syntheticItems(30),
deletedExternalIds: [],
syncToken: null,
complete: true,
});
expect(pushed.created).toBe(30);
await rpc(page, "media/collections/recluster", {});
await expect
.poll(
async () =>
(
await rpc<{ items: unknown[] }>(page, "media/collections/list", {
kind: "session",
cursor: null,
limit: 10,
})
).items.length,
{ timeout: 90_000, intervals: [1000] },
)
.toBeGreaterThan(0);
const studio = await openMediaStudio(page);
const sessions = studio.getByTestId("media-collection-card");
await expect(sessions.first()).toBeVisible();
await expect(studio.getByTestId("media-indexing")).toHaveCount(0);
await captureScreenshot(page, testInfo, "media-library-sessions");
await studio.getByRole("tab", { name: "Themes" }).click();
await expect(studio.getByRole("tab", { name: "Themes" })).toHaveAttribute(
"aria-selected",
"true",
);
await captureScreenshot(page, testInfo, "media-library-themes");
await studio.getByRole("tab", { name: "Sessions" }).click();
await studio.getByRole("button", { name: "Videos", exact: true }).click();
const grid = studio.getByTestId("media-asset-grid");
await expect(grid).toBeVisible();
await expect(grid.getByTestId("media-asset-tile").first()).toBeVisible();
await expect(grid).toHaveAttribute("data-count", "24");
await captureScreenshot(page, testInfo, "media-library-filter-videos");
await studio.getByRole("button", { name: "Videos", exact: true }).click();
await studio.getByTestId("media-collection-card").first().click();
await expect(studio).toHaveAttribute("data-view", "collection");
const tiles = studio.getByTestId("media-asset-tile");
await expect(tiles.first()).toBeVisible();
await tiles.nth(0).click();
await tiles.nth(1).click();
await expect(tiles.nth(0)).toHaveAttribute("aria-pressed", "true");
await expect(studio.getByRole("button", { name: "2 selected" })).toBeVisible();
await captureScreenshot(page, testInfo, "media-collection-selection");
await studio.getByTestId("media-create-project-open").click();
const create = page.getByTestId("media-create-project");
await expect(create).toBeVisible();
await expect(create.getByRole("checkbox").first()).toBeVisible();
await captureScreenshot(page, testInfo, "media-create-project");
await create.getByRole("button", { name: "Create project" }).click();
await expect(studio).toHaveAttribute("data-view", "project");
await studio.getByRole("button", { name: "Back" }).click();
await expect(studio).toHaveAttribute("data-view", "collection");
});
test("people view confirms, merges and records consent", async ({ page }, testInfo) => {
const stamp = Date.now();
await signup(page, `media-people-${stamp}@rakazo.test`, "password12", "Media People");
await completeOnboarding(page);
let people = [
person("person-a", "Person 1"),
person("person-b", "Person 2", { confirmed: true }),
person("person-c", "Person 3"),
];
const consentCalls: Json[] = [];
await stub(page, "media/people/list", () => people);
await stub(page, "media/people/update", (input) => {
const { personId, ...patch } = input;
people = people.map((row) => (row.id === personId ? { ...row, ...patch } : row));
return people.find((row) => row.id === personId);
});
await stub(page, "media/people/merge", (input) => {
const [target, ...others] = input.personIds as string[];
people = people
.filter((row) => !others.includes(row.id))
.map((row) => (row.id === target ? { ...row, confirmed: true, faceTrackCount: 6 } : row));
return people.find((row) => row.id === target);
});
await stub(page, "media/people/recordConsent", (input) => {
consentCalls.push(input);
people = people.map((row) =>
row.id === input.personId
? {
...row,
consentRecordedAt: input.consentRecordedAt,
ageVerificationMethod: input.ageVerificationMethod,
ageVerifiedAt: input.ageVerifiedAt,
}
: row,
);
return people.find((row) => row.id === input.personId);
});
const studio = await openMediaStudio(page);
await studio.getByTestId("media-connect-source-open").click();
const connect = page.getByTestId("media-connect-source");
await expect(connect).toBeVisible();
await captureScreenshot(page, testInfo, "media-library-connect-source");
await page.keyboard.press("Escape");
await expect(connect).toBeHidden();
await expect(studio).toBeVisible();
await studio.getByRole("tab", { name: "People" }).click();
await expect(studio).toHaveAttribute("data-view", "people");
const cards = studio.getByTestId("media-person");
await expect(cards).toHaveCount(3);
await captureScreenshot(page, testInfo, "media-people");
const first = cards.filter({ hasText: "Person 1" });
await first.getByRole("button", { name: "Confirm" }).click();
await expect(first.getByText("Confirmed")).toBeVisible();
await first.getByRole("combobox", { name: "Anonymization" }).selectOption("blur");
await expect(first.getByRole("combobox", { name: "Anonymization" })).toHaveValue("blur");
await expect(
first
.getByRole("combobox", { name: "Anonymization" })
.locator("option[value=synthetic_identity]"),
).toBeDisabled();
await first.getByRole("button", { name: "Add consent" }).click();
const consent = page.getByTestId("media-consent");
await expect(consent).toBeVisible();
await consent.getByLabel("Consent given on").fill("2026-09-01");
await consent.getByLabel("Age proof").selectOption("platform_verified");
await consent.getByLabel("Verified on").fill("2026-09-02");
await captureScreenshot(page, testInfo, "media-people-consent");
await consent.getByRole("button", { name: "Save" }).click();
await expect(consent).toBeHidden();
expect(consentCalls).toEqual([
{
personId: "person-a",
consentArtifactId: null,
consentRecordedAt: "2026-09-01T12:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-02T12:00:00.000Z",
},
]);
await expect(first.locator("[data-consent=complete]")).toBeVisible();
await cards
.filter({ hasText: "Person 2" })
.getByRole("checkbox", { name: "Select Person 2" })
.click();
await cards
.filter({ hasText: "Person 3" })
.getByRole("checkbox", { name: "Select Person 3" })
.click();
await studio.getByRole("button", { name: "Merge" }).click();
await page.getByRole("alertdialog").getByRole("button", { name: "Merge" }).click();
await expect(cards).toHaveCount(2);
});
test("results import a CSV with a mapping and show the funnel", async ({ page }, testInfo) => {
const stamp = Date.now();
await signup(page, `media-results-${stamp}@rakazo.test`, "password12", "Media Results");
await completeOnboarding(page);
let imported = false;
const saved: Json[] = [];
await stub(page, "media/metrics/funnel", () =>
imported
? [
{
publicationId: "pub-1",
projectId: "project-1",
profileId: "example-short-vertical",
variant: "15s",
thumbnailVariant: "a",
views: 12000,
clicks: 840,
conversions: 42,
revenueCents: 41958,
currency: "EUR",
},
{
publicationId: "pub-2",
projectId: "project-1",
profileId: "example-paid-video",
variant: null,
thumbnailVariant: null,
views: 3000,
clicks: 300,
conversions: 30,
revenueCents: 29970,
currency: "EUR",
},
]
: [],
);
await stub(page, "media/metrics/segments", () =>
imported
? [
{
segmentId: "segment-1",
assetId: "asset-1",
startMs: 1000,
endMs: 4000,
publications: 2,
views: 15000,
clicks: 1140,
conversions: 72,
revenueCents: 71928,
currency: "EUR",
},
]
: [],
);
await stub(page, "media/publications/list", () => ({ items: [], nextCursor: null }));
await stub(page, "media/metrics/mappings", () => []);
await stub(page, "media/metrics/saveMapping", (input) => {
saved.push(input);
return { ...input, id: "mapping-1" };
});
await stub(page, "media/metrics/importCsv", () => {
imported = true;
return { imported: 2, skipped: [] };
});
const studio = await openMediaStudio(page);
await studio.getByRole("tab", { name: "Results" }).click();
await expect(studio.getByText("No results yet.")).toBeVisible();
await studio.getByRole("button", { name: "Import CSV" }).click();
const dialog = page.getByTestId("media-csv-import");
await dialog.getByLabel("CSV file").setInputFiles({
name: "example-platform.csv",
mimeType: "text/csv",
buffer: Buffer.from(
"date;utm_content;views;clicks;purchases;revenue\n2026-09-01;variant-a;12000;840;42;419.58\n2026-09-01;variant-b;3000;300;30;299.70\n",
),
});
await expect(dialog.getByLabel("Views")).toHaveValue("views");
await expect(dialog.getByLabel("Purchases")).toHaveValue("purchases");
await captureScreenshot(page, testInfo, "media-results-mapping");
await dialog.getByRole("button", { name: /^Import/ }).click();
await expect(dialog.getByTestId("media-csv-imported")).toHaveText("2 rows imported");
expect(saved[0]).toMatchObject({
name: "example-platform",
delimiter: ";",
publicationKey: "utm_content",
columns: { publication: "utm_content", observedAt: "date", views: "views" },
});
await dialog.getByRole("button", { name: "Done" }).click();
const funnel = studio.getByTestId("media-funnel");
await expect(funnel).toBeVisible();
await expect(funnel).toContainText("15,000");
await expect(funnel).toContainText("1,140");
await expect(funnel).toContainText("72");
await expect(studio.getByTestId("media-results-segments")).toBeVisible();
await captureScreenshot(page, testInfo, "media-results-funnel");
await studio.getByRole("button", { name: "More" }).click();
await page.getByRole("menuitem", { name: "Reset teaser model" }).click();
await page.getByRole("alertdialog").getByRole("button", { name: "Reset" }).click();
await expect(studio.getByText("Reset", { exact: true })).toBeVisible();
});
test("the asset grid stays virtualized across 50k assets", async ({ page }) => {
test.setTimeout(240_000);
const stamp = Date.now();
await signup(page, `media-scale-${stamp}@rakazo.test`, "password12", "Media Scale");
await completeOnboarding(page);
const total = 50_000;
await stub(page, "media/sources/list", () => [
{
id: "scale-source",
kind: "folder",
label: "Archive",
status: "idle",
assetCount: total,
lastSyncedAt: now,
error: null,
createdAt: now,
},
]);
await stub(page, "media/assets/search", (input) => {
if (Array.isArray(input.ingestStates)) return { items: [], nextCursor: null };
const offset = Number(input.cursor ?? 0);
const limit = Number(input.limit ?? 100);
const end = Math.min(total, offset + limit);
return {
items: Array.from({ length: end - offset }, (_, index) => syntheticAsset(offset + index)),
nextCursor: end < total ? String(end) : null,
};
});
const studio = await openMediaStudio(page);
await studio.getByRole("button", { name: "Videos", exact: true }).click();
const grid = studio.getByTestId("media-asset-grid");
await expect(grid.getByTestId("media-asset-tile").first()).toBeVisible();
let loaded = 0;
while (loaded < total) {
await grid.evaluate((element) => {
element.scrollTop = element.scrollHeight;
});
await expect
.poll(async () => Number(await grid.getAttribute("data-count")), { timeout: 10_000 })
.toBeGreaterThan(loaded);
loaded = Number(await grid.getAttribute("data-count"));
expect(await grid.getByTestId("media-asset-tile").count()).toBeLessThan(400);
}
expect(loaded).toBe(total);
await grid.evaluate((element) => {
element.scrollTop = element.scrollHeight;
});
await expect(grid.locator('[data-asset-id="scale-49999"]')).toBeVisible();
expect(await grid.getByTestId("media-asset-tile").count()).toBeLessThan(400);
});

View File

@ -24,6 +24,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/ui-tokens": "workspace:*",
"@rakazo/ui-web": "workspace:*",
"@tanstack/react-virtual": "3.14.13",
"better-auth": "1.7.4",
"lucide-react": "1.44.0",
"react": "19.2.3",

View File

@ -0,0 +1,80 @@
import { useLingui } from "@lingui/react/macro";
import type { MediaAsset } from "@rakazo/contracts";
import type { ReactNode } from "react";
import { useCallback, useRef, useState } from "react";
import { AssetTile } from "./AssetTile";
import { VirtualGrid } from "./VirtualGrid";
/** Selected asset ids; shift-click selects the range since the last click. */
export function useAssetSelection() {
const [selected, setSelected] = useState<ReadonlySet<string>>(() => new Set());
const anchor = useRef<number | null>(null);
const toggle = useCallback((assets: readonly MediaAsset[], index: number, range: boolean) => {
const target = assets[index];
if (!target) return;
const start = anchor.current;
setSelected((current) => {
const next = new Set(current);
if (range && start !== null) {
const [from, to] = [Math.min(start, index), Math.max(start, index)];
for (const asset of assets.slice(from, to + 1)) next.add(asset.id);
} else if (next.has(target.id)) {
next.delete(target.id);
} else {
next.add(target.id);
}
return next;
});
anchor.current = index;
}, []);
const clear = useCallback(() => {
anchor.current = null;
setSelected(new Set());
}, []);
return { selected, toggle, clear };
}
export function AssetGrid({
assets,
selection,
scrub = false,
onNearEnd,
footer,
}: {
assets: readonly MediaAsset[];
selection: ReturnType<typeof useAssetSelection>;
scrub?: boolean;
onNearEnd?: () => void;
footer?: ReactNode;
}) {
const { t } = useLingui();
const selecting = selection.selected.size > 0;
return (
<VirtualGrid
testId="media-asset-grid"
label={t`Clips`}
count={assets.length}
getKey={(index) => assets[index]?.id ?? index}
minTileWidth={150}
tileAspect={9 / 16}
onNearEnd={onNearEnd}
footer={footer}
renderItem={(index) => {
const asset = assets[index];
if (!asset) return null;
return (
<AssetTile
asset={asset}
scrub={scrub}
selecting={selecting}
selected={selection.selected.has(asset.id)}
onToggle={(event) => selection.toggle(assets, index, event.shiftKey)}
/>
);
}}
/>
);
}

View File

@ -0,0 +1,155 @@
import { useLingui } from "@lingui/react/macro";
import type { MediaAsset } from "@rakazo/contracts";
import { Checkbox, cn } from "@rakazo/ui-web";
import { Image as ImageIcon, Star, Video } from "lucide-react";
import { useEffect, useRef, useState } from "react";
import { mediaAssetPath, useMediaObjectUrl } from "../../../lib/media-bytes";
import { formatDuration, scrubFraction } from "../../../lib/media-library";
/** Poster of an asset, or a quiet placeholder until the proxy job made one. */
export function AssetThumb({
assetId,
hasThumbnail,
kind,
className,
}: {
assetId: string | null;
hasThumbnail: boolean;
kind?: MediaAsset["kind"];
className?: string;
}) {
const thumb = useMediaObjectUrl(
assetId && hasThumbnail ? mediaAssetPath(assetId, "thumb") : null,
{ keep: true },
);
if (thumb.url) {
return (
<img
src={thumb.url}
alt=""
draggable={false}
className={cn("size-full object-cover", className)}
/>
);
}
const Icon = kind === "photo" || kind === "live_photo" ? ImageIcon : Video;
return (
<div
className={cn(
"grid size-full place-items-center bg-muted text-muted-foreground/60",
thumb.status === "loading" && "animate-pulse",
className,
)}
>
<Icon strokeWidth={1.6} className="size-icon-lg" aria-hidden="true" />
</div>
);
}
/** Plays the proxy under the pointer: left edge is the first frame, right edge the last. */
function Scrubber({ asset, fraction }: { asset: MediaAsset; fraction: number }) {
const proxy = useMediaObjectUrl(mediaAssetPath(asset.id, "proxy"));
const videoRef = useRef<HTMLVideoElement>(null);
const duration = (asset.durationMs ?? 0) / 1000;
useEffect(() => {
const video = videoRef.current;
if (!proxy.url || !video || duration <= 0) return;
const target = fraction * duration;
if (Math.abs(video.currentTime - target) > 0.04) video.currentTime = target;
}, [proxy.url, fraction, duration]);
return (
<>
{proxy.url ? (
<video
ref={videoRef}
src={proxy.url}
muted
playsInline
preload="auto"
className="absolute inset-0 size-full object-cover"
onLoadedMetadata={(event) => {
event.currentTarget.currentTime = fraction * (event.currentTarget.duration || 0);
}}
/>
) : null}
<span
aria-hidden="true"
className="absolute bottom-0 left-0 h-0.5 bg-primary-foreground"
style={{ width: `${fraction * 100}%` }}
/>
</>
);
}
export function AssetTile({
asset,
selected,
selecting,
onToggle,
scrub = false,
}: {
asset: MediaAsset;
selected: boolean;
/** Show checkboxes on every tile, not only on hover. */
selecting: boolean;
onToggle: (event: React.MouseEvent | React.KeyboardEvent) => void;
scrub?: boolean;
}) {
const { t } = useLingui();
const [fraction, setFraction] = useState<number | null>(null);
const canScrub = scrub && asset.kind !== "photo" && asset.hasProxy && (asset.durationMs ?? 0) > 0;
const duration = asset.kind === "photo" ? "" : formatDuration(asset.durationMs);
const captured = asset.capturedAt ? new Date(asset.capturedAt).toLocaleString() : "";
return (
<button
type="button"
data-testid="media-asset-tile"
data-asset-id={asset.id}
aria-pressed={selected}
aria-label={captured || t`Clip`}
onClick={onToggle}
onPointerMove={
canScrub
? (event) => {
const box = event.currentTarget.getBoundingClientRect();
setFraction(scrubFraction(event.clientX, box.left, box.width));
}
: undefined
}
onPointerLeave={canScrub ? () => setFraction(null) : undefined}
className={cn(
"group/tile relative size-full overflow-hidden rounded-lg bg-muted outline-none ring-offset-2 ring-offset-background focus-visible:ring-2 focus-visible:ring-ring",
selected && "ring-2 ring-primary",
)}
>
<AssetThumb assetId={asset.id} hasThumbnail={asset.hasThumbnail} kind={asset.kind} />
{canScrub && fraction !== null ? <Scrubber asset={asset} fraction={fraction} /> : null}
<span
className={cn(
"absolute left-1.5 top-1.5 transition-opacity",
selecting || selected ? "opacity-100" : "opacity-0 group-hover/tile:opacity-100",
)}
>
<Checkbox
checked={selected}
tabIndex={-1}
aria-hidden="true"
className="pointer-events-none bg-background/80"
/>
</span>
{asset.isFavorite ? (
<Star
aria-hidden="true"
fill="currentColor"
className="absolute right-1.5 top-1.5 size-icon-sm text-primary-foreground drop-shadow"
/>
) : null}
{duration ? (
<span className="absolute bottom-1.5 right-1.5 rounded bg-foreground/70 px-1 font-mono text-ui-caption tabular-nums text-background">
{duration}
</span>
) : null}
</button>
);
}

View File

@ -0,0 +1,57 @@
import { Trans } from "@lingui/react/macro";
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@rakazo/ui-web";
import type { ReactNode } from "react";
export function ConfirmDialog({
open,
title,
description,
action,
destructive = false,
onConfirm,
onCancel,
}: {
open: boolean;
title: ReactNode;
description?: ReactNode;
action: ReactNode;
destructive?: boolean;
onConfirm: () => void;
onCancel: () => void;
}) {
return (
<AlertDialog
open={open}
onOpenChange={(next) => {
if (!next) onCancel();
}}
>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>{title}</AlertDialogTitle>
{description ? <AlertDialogDescription>{description}</AlertDialogDescription> : null}
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>
<Trans>Cancel</Trans>
</AlertDialogCancel>
<AlertDialogAction
variant={destructive ? "destructive" : "default"}
onClick={() => onConfirm()}
>
{action}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
);
}

View File

@ -0,0 +1,248 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { MediaSource } from "@rakazo/contracts";
import {
Button,
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
Field,
FieldLabel,
Input,
} from "@rakazo/ui-web";
import { FolderOpen, Images, RefreshCw, Trash2 } from "lucide-react";
import { useId, useState } from "react";
import { desktopBridge } from "../../../lib/desktop";
import { rpcErrorMessage } from "../../../lib/media-studio";
import { rpc } from "../../../lib/rpc";
import { ConfirmDialog } from "./ConfirmDialog";
import type { usePhotosSync } from "./usePhotosSync";
const SERVER_SYNCED = new Set<MediaSource["kind"]>(["folder", "emulator"]);
export function ConnectSourceDialog({
open,
onOpenChange,
sources,
canLinkFolder,
photos,
onChanged,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
sources: readonly MediaSource[];
canLinkFolder: boolean;
photos: ReturnType<typeof usePhotosSync>;
onChanged: () => void;
}) {
const { t } = useLingui();
const labelId = useId();
const pathId = useId();
const [label, setLabel] = useState("");
const [path, setPath] = useState("");
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const [removing, setRemoving] = useState<MediaSource | null>(null);
const picker = desktopBridge()?.files?.pickDirectory;
const nothingToConnect = !canLinkFolder && !photos.available;
const run = async (action: () => Promise<unknown>) => {
setBusy(true);
setError(null);
try {
await action();
onChanged();
return true;
} catch (cause) {
setError(rpcErrorMessage(cause, t`That did not work. Try again.`));
return false;
} finally {
setBusy(false);
}
};
const connectFolder = async () => {
const ok = await run(() =>
rpc.media.sources.createFolder({
label: label.trim() || folderName(path) || t`Folder`,
path: path.trim(),
}),
);
if (ok) {
setLabel("");
setPath("");
onOpenChange(false);
}
};
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent data-testid="media-connect-source" className="sm:max-w-md">
<DialogHeader>
<DialogTitle>
<Trans>Sources</Trans>
</DialogTitle>
</DialogHeader>
{sources.length > 0 ? (
<ul className="flex flex-col divide-y divide-border rounded-lg border border-border">
{sources.map((source) => (
<li key={source.id} className="flex items-center gap-2 px-3 py-2">
<div className="min-w-0 flex-1">
<div className="truncate text-ui-body-sm font-medium" dir="auto">
{source.label}
</div>
<div className="truncate text-ui-caption text-muted-foreground">
{source.status === "error" && source.error ? (
<span className="text-destructive">{source.error}</span>
) : (
<Trans>{source.assetCount} items</Trans>
)}
</div>
</div>
{SERVER_SYNCED.has(source.kind) ? (
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Sync`}
disabled={busy || source.status === "syncing"}
onClick={() => void run(() => rpc.media.sources.sync({ sourceId: source.id }))}
>
<RefreshCw className={source.status === "syncing" ? "animate-spin" : ""} />
</Button>
) : null}
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Remove ${source.label}`}
disabled={busy}
onClick={() => setRemoving(source)}
>
<Trash2 />
</Button>
</li>
))}
</ul>
) : null}
{photos.available ? (
<Button
variant="outline"
className="justify-start"
disabled={photos.syncing || photos.denied}
onClick={() => {
void photos.sync();
onOpenChange(false);
}}
>
<Images aria-hidden="true" />
<Trans>Apple Photos</Trans>
</Button>
) : null}
{photos.denied ? (
<p className="text-ui-body-sm text-muted-foreground">
<Trans>Allow evOo to access Photos in System Settings.</Trans>
</p>
) : null}
{canLinkFolder ? (
<form
className="flex flex-col gap-3"
onSubmit={(event) => {
event.preventDefault();
void connectFolder();
}}
>
<Field>
<FieldLabel htmlFor={pathId}>
<Trans>Folder</Trans>
</FieldLabel>
<div className="flex gap-2">
<Input
id={pathId}
value={path}
onChange={(event) => setPath(event.target.value)}
placeholder={t`/Users/you/Movies`}
spellCheck={false}
autoComplete="off"
/>
{picker ? (
<Button
type="button"
variant="outline"
aria-label={t`Choose folder`}
onClick={() => {
void picker()
.then((picked) => {
if (picked) setPath(picked);
})
.catch(() => undefined);
}}
>
<FolderOpen aria-hidden="true" />
</Button>
) : null}
</div>
</Field>
{path.trim() ? (
<Field>
<FieldLabel htmlFor={labelId}>
<Trans>Name</Trans>
</FieldLabel>
<Input
id={labelId}
value={label}
maxLength={60}
onChange={(event) => setLabel(event.target.value)}
placeholder={folderName(path)}
/>
</Field>
) : null}
<DialogFooter>
<Button type="submit" disabled={busy || !path.trim()}>
<Trans>Connect</Trans>
</Button>
</DialogFooter>
</form>
) : null}
{nothingToConnect && sources.length === 0 ? (
<p className="text-ui-body-sm text-muted-foreground">
<Trans>Only the owner of this evOo can connect a folder.</Trans>
</p>
) : null}
{error || photos.error ? (
<p role="alert" className="text-ui-body-sm text-destructive">
{error ?? photos.error}
</p>
) : null}
<ConfirmDialog
open={removing !== null}
destructive
title={<Trans>Remove {removing?.label ?? ""}?</Trans>}
description={
<Trans>Its clips leave the library. Your original files stay untouched.</Trans>
}
action={<Trans>Remove</Trans>}
onCancel={() => setRemoving(null)}
onConfirm={() => {
const target = removing;
setRemoving(null);
if (target) void run(() => rpc.media.sources.remove({ sourceId: target.id }));
}}
/>
</DialogContent>
</Dialog>
);
}
function folderName(path: string): string {
return (
path
.trim()
.replace(/[\\/]+$/, "")
.split(/[\\/]/)
.at(-1) ?? ""
);
}

View File

@ -0,0 +1,151 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { AgeVerificationMethod, MediaPerson } from "@rakazo/contracts";
import {
Button,
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
Field,
FieldLabel,
Input,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { useEffect, useId, useState } from "react";
import type { ConsentForm } from "../../../lib/media-library";
import { consentInput, isoToDay, todayDay } from "../../../lib/media-library";
import { rpcErrorMessage } from "../../../lib/media-studio";
import { rpc } from "../../../lib/rpc";
function initialForm(person: MediaPerson): ConsentForm {
return {
consentDate: isoToDay(person.consentRecordedAt) || todayDay(),
method: person.ageVerificationMethod,
verifiedDate: isoToDay(person.ageVerifiedAt) || todayDay(),
};
}
/** The only place consent and age proof are written; agents never reach it. */
export function ConsentDialog({
person,
onOpenChange,
onSaved,
}: {
person: MediaPerson | null;
onOpenChange: (open: boolean) => void;
onSaved: (person: MediaPerson) => void;
}) {
const { t } = useLingui();
const consentId = useId();
const methodId = useId();
const verifiedId = useId();
const [form, setForm] = useState<ConsentForm | null>(null);
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
setForm(person ? initialForm(person) : null);
setError(null);
}, [person]);
const methods: Record<AgeVerificationMethod, string> = {
platform_verified: t`Verified by platform`,
document_on_file: t`ID document on file`,
none: t`No age proof`,
};
const input = person && form ? consentInput(person.id, form) : null;
const today = todayDay();
const save = async () => {
if (!input) return;
setBusy(true);
setError(null);
try {
onSaved(await rpc.media.people.recordConsent(input));
onOpenChange(false);
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not save consent.`));
} finally {
setBusy(false);
}
};
return (
<Dialog open={person !== null} onOpenChange={onOpenChange}>
<DialogContent data-testid="media-consent" className="sm:max-w-sm">
<DialogHeader>
<DialogTitle>
<Trans>Consent · {person?.displayName ?? ""}</Trans>
</DialogTitle>
</DialogHeader>
{form ? (
<form
className="flex flex-col gap-4"
onSubmit={(event) => {
event.preventDefault();
void save();
}}
>
<Field>
<FieldLabel htmlFor={consentId}>
<Trans>Consent given on</Trans>
</FieldLabel>
<Input
id={consentId}
type="date"
max={today}
value={form.consentDate}
onChange={(event) => setForm({ ...form, consentDate: event.target.value })}
/>
</Field>
<Field>
<FieldLabel htmlFor={methodId}>
<Trans>Age proof</Trans>
</FieldLabel>
<NativeSelect
id={methodId}
className="w-full"
value={form.method}
onChange={(event) =>
setForm({ ...form, method: event.target.value as AgeVerificationMethod })
}
>
{(Object.keys(methods) as AgeVerificationMethod[]).map((method) => (
<NativeSelectOption key={method} value={method}>
{methods[method]}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
{form.method !== "none" ? (
<Field>
<FieldLabel htmlFor={verifiedId}>
<Trans>Verified on</Trans>
</FieldLabel>
<Input
id={verifiedId}
type="date"
max={today}
value={form.verifiedDate}
onChange={(event) => setForm({ ...form, verifiedDate: event.target.value })}
/>
</Field>
) : null}
{error ? (
<p role="alert" className="text-ui-body-sm text-destructive">
{error}
</p>
) : null}
<DialogFooter>
<Button type="submit" disabled={busy || !input}>
<Trans>Save</Trans>
</Button>
</DialogFooter>
</form>
) : null}
</DialogContent>
</Dialog>
);
}

View File

@ -0,0 +1,208 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { MediaProjectGoal, PlatformProfileSummary } from "@rakazo/contracts";
import { MediaProjectGoalSchema } from "@rakazo/contracts";
import {
Badge,
Button,
Checkbox,
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
Field,
FieldLabel,
FieldLegend,
FieldSet,
Input,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { useEffect, useId, useState } from "react";
import { rpcErrorMessage } from "../../../lib/media-studio";
import { rpc } from "../../../lib/rpc";
export function useGoalLabels(): Record<MediaProjectGoal, string> {
const { t } = useLingui();
return {
teaser: t`Teaser`,
full_release: t`Full release`,
trailer: t`Trailer`,
highlight: t`Highlight`,
story: t`Story`,
custom: t`Custom`,
};
}
export function CreateProjectDialog({
open,
onOpenChange,
defaultTitle,
collectionIds,
assetIds,
onCreated,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
defaultTitle: string;
collectionIds: string[];
assetIds: string[];
onCreated: (projectId: string) => void;
}) {
const { t } = useLingui();
const goals = useGoalLabels();
const titleId = useId();
const goalId = useId();
const [title, setTitle] = useState(defaultTitle);
const [goal, setGoal] = useState<MediaProjectGoal>("teaser");
const [profiles, setProfiles] = useState<PlatformProfileSummary[] | null>(null);
const [chosen, setChosen] = useState<ReadonlySet<string>>(() => new Set());
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
if (!open) return;
setTitle(defaultTitle);
setError(null);
let active = true;
rpc.media.profiles
.list({})
.then((list) => {
if (!active) return;
setProfiles(list);
setChosen((current) =>
current.size > 0 ? current : new Set(list.slice(0, 1).map((p) => p.id)),
);
})
.catch((cause) => {
if (active) setError(rpcErrorMessage(cause, t`Could not load platforms.`));
});
return () => {
active = false;
};
}, [open, defaultTitle, t]);
const create = async () => {
setBusy(true);
setError(null);
try {
const project = await rpc.media.projects.create({
title: title.trim(),
goal,
collectionIds,
assetIds,
profileIds: [...chosen],
brandKitId: null,
});
onOpenChange(false);
onCreated(project.id);
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not create the project.`));
} finally {
setBusy(false);
}
};
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent data-testid="media-create-project" className="sm:max-w-md">
<DialogHeader>
<DialogTitle>
<Trans>New project</Trans>
</DialogTitle>
</DialogHeader>
<form
className="flex flex-col gap-4"
onSubmit={(event) => {
event.preventDefault();
void create();
}}
>
<Field>
<FieldLabel htmlFor={titleId}>
<Trans>Title</Trans>
</FieldLabel>
<Input
id={titleId}
value={title}
maxLength={120}
onChange={(event) => setTitle(event.target.value)}
/>
</Field>
<Field>
<FieldLabel htmlFor={goalId}>
<Trans>Goal</Trans>
</FieldLabel>
<NativeSelect
id={goalId}
className="w-full"
value={goal}
onChange={(event) => setGoal(event.target.value as MediaProjectGoal)}
>
{MediaProjectGoalSchema.options.map((option) => (
<NativeSelectOption key={option} value={option}>
{goals[option]}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
<FieldSet>
<FieldLegend variant="label">
<Trans>Platforms</Trans>
</FieldLegend>
{profiles?.length === 0 ? (
<p className="text-ui-body-sm text-muted-foreground">
<Trans>No platform profiles found.</Trans>
</p>
) : null}
<div className="flex max-h-56 flex-col gap-1 overflow-y-auto">
{(profiles ?? []).map((profile) => (
<div
key={profile.id}
className="flex items-center gap-2.5 rounded-md px-1 py-1.5 text-ui-body-sm hover:bg-muted"
>
<Checkbox
id={`${goalId}-${profile.id}`}
checked={chosen.has(profile.id)}
onCheckedChange={(checked) =>
setChosen((current) => {
const next = new Set(current);
if (checked) next.add(profile.id);
else next.delete(profile.id);
return next;
})
}
/>
<label
htmlFor={`${goalId}-${profile.id}`}
className="min-w-0 flex-1 cursor-pointer truncate"
>
{profile.name}
</label>
<span className="text-ui-caption text-muted-foreground">
{profile.aspectRatios.join(" · ")}
</span>
{profile.unverified || profile.stale ? (
<Badge variant="outline">
<Trans>Unverified</Trans>
</Badge>
) : null}
</div>
))}
</div>
</FieldSet>
{error ? (
<p role="alert" className="text-ui-body-sm text-destructive">
{error}
</p>
) : null}
<DialogFooter>
<Button type="submit" disabled={busy || !title.trim() || chosen.size === 0}>
<Trans>Create project</Trans>
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
);
}

View File

@ -0,0 +1,354 @@
import { Plural, Trans, useLingui } from "@lingui/react/macro";
import type { MetricCsvMapping } from "@rakazo/contracts";
import {
Button,
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
Field,
FieldLabel,
Input,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { useEffect, useId, useMemo, useState } from "react";
import type { CsvDelimiter, MappingColumn } from "../../../lib/media-library";
import {
csvHeaders,
firstCsvLine,
guessColumns,
guessDelimiter,
guessPublicationKey,
MAPPING_COLUMNS,
} from "../../../lib/media-library";
import { rpcErrorMessage } from "../../../lib/media-studio";
import { rpc } from "../../../lib/rpc";
const MAX_CSV_CHARS = 5_000_000;
const NEW = "";
type Draft = Omit<MetricCsvMapping, "columns"> & {
columns: Partial<Record<MappingColumn, string>>;
};
function draftFrom(mapping: MetricCsvMapping): Draft {
return { ...mapping, columns: { ...mapping.columns } };
}
function toMapping(draft: Draft): MetricCsvMapping | null {
const { publication, observedAt } = draft.columns;
if (!publication || !observedAt || !draft.name.trim()) return null;
if (!/^[A-Za-z]{3}$/.test(draft.defaultCurrency)) return null;
const columns: MetricCsvMapping["columns"] = { publication, observedAt };
for (const key of MAPPING_COLUMNS) {
const value = draft.columns[key];
if (value) columns[key] = value;
}
return {
...draft,
name: draft.name.trim(),
defaultCurrency: draft.defaultCurrency.toUpperCase(),
columns,
};
}
export function CsvImportDialog({
open,
onOpenChange,
onImported,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
onImported: () => void;
}) {
const { t } = useLingui();
const fileId = useId();
const mappingId = useId();
const nameId = useId();
const [mappings, setMappings] = useState<MetricCsvMapping[]>([]);
const [csv, setCsv] = useState<string | null>(null);
const [fileName, setFileName] = useState("");
const [draft, setDraft] = useState<Draft | null>(null);
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const [result, setResult] = useState<{
imported: number;
skipped: Array<{ row: number; reason: string }>;
} | null>(null);
useEffect(() => {
if (!open) return;
setCsv(null);
setFileName("");
setDraft(null);
setError(null);
setResult(null);
rpc.media.metrics
.mappings({})
.then(setMappings)
.catch(() => setMappings([]));
}, [open]);
const headers = useMemo(
() => (csv && draft ? csvHeaders(csv, draft.delimiter) : []),
[csv, draft?.delimiter, draft],
);
const labels: Record<MappingColumn, string> = {
publication: t`Publication`,
observedAt: t`Date`,
views: t`Views`,
clicks: t`Clicks`,
conversions: t`Purchases`,
revenue: t`Revenue`,
currency: t`Currency`,
};
const keyLabels: Record<MetricCsvMapping["publicationKey"], string> = {
utm_content: t`UTM content`,
tracking_url: t`Tracking link`,
external_url: t`Post link`,
id: t`Publication ID`,
};
const readFile = async (file: File) => {
setError(null);
setResult(null);
if (file.size > MAX_CSV_CHARS) {
setError(t`This file is too large.`);
return;
}
const text = await file.text();
setCsv(text);
setFileName(file.name);
const delimiter: CsvDelimiter = guessDelimiter(firstCsvLine(text));
const found = csvHeaders(text, delimiter);
const matching = mappings.find((mapping) =>
Object.values(mapping.columns).every((column) => !column || found.includes(column)),
);
if (matching) {
setDraft(draftFrom(matching));
return;
}
const columns = guessColumns(found);
setDraft({
name: file.name.replace(/\.[^.]+$/, "").slice(0, 60) || t`Import`,
delimiter,
columns,
publicationKey: guessPublicationKey(columns.publication),
revenueUnit: "major",
defaultCurrency: "EUR",
});
};
const mapping = draft ? toMapping(draft) : null;
const runImport = async () => {
if (!csv || !mapping) return;
setBusy(true);
setError(null);
try {
const saved = await rpc.media.metrics.saveMapping(mapping);
if (!saved.id) throw new Error(t`Could not save the mapping.`);
const outcome = await rpc.media.metrics.importCsv({ mappingId: saved.id, csv });
setResult(outcome);
onImported();
} catch (cause) {
setError(rpcErrorMessage(cause, t`Import failed.`));
} finally {
setBusy(false);
}
};
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent data-testid="media-csv-import" className="sm:max-w-lg">
<DialogHeader>
<DialogTitle>
<Trans>Import results</Trans>
</DialogTitle>
</DialogHeader>
{result ? (
<div className="flex flex-col gap-2 text-ui-body-sm">
<p data-testid="media-csv-imported">
<Plural value={result.imported} one="# row imported" other="# rows imported" />
</p>
{result.skipped.length > 0 ? (
<ul className="max-h-40 overflow-y-auto text-muted-foreground">
{result.skipped.slice(0, 20).map((skip) => (
<li key={skip.row}>
<Trans>
Row {skip.row}: {skip.reason}
</Trans>
</li>
))}
</ul>
) : null}
<DialogFooter>
<Button onClick={() => onOpenChange(false)}>
<Trans>Done</Trans>
</Button>
</DialogFooter>
</div>
) : (
<form
className="flex flex-col gap-4"
onSubmit={(event) => {
event.preventDefault();
void runImport();
}}
>
<Field>
<FieldLabel htmlFor={fileId}>
<Trans>CSV file</Trans>
</FieldLabel>
<Input
id={fileId}
type="file"
accept=".csv,text/csv,text/plain"
onChange={(event) => {
const file = event.target.files?.[0];
if (file) void readFile(file);
}}
/>
</Field>
{draft && csv ? (
<>
{mappings.length > 0 ? (
<Field>
<FieldLabel htmlFor={mappingId}>
<Trans>Mapping</Trans>
</FieldLabel>
<NativeSelect
id={mappingId}
className="w-full"
value={draft.id ?? NEW}
onChange={(event) => {
const chosen = mappings.find((row) => row.id === event.target.value);
setDraft(chosen ? draftFrom(chosen) : { ...draft, id: undefined });
}}
>
<NativeSelectOption value={NEW}>{t`New mapping`}</NativeSelectOption>
{mappings.map((row) => (
<NativeSelectOption key={row.id} value={row.id ?? NEW}>
{row.name}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
) : null}
<Field>
<FieldLabel htmlFor={nameId}>
<Trans>Platform</Trans>
</FieldLabel>
<Input
id={nameId}
value={draft.name}
maxLength={60}
onChange={(event) => setDraft({ ...draft, name: event.target.value })}
/>
</Field>
<div className="grid grid-cols-2 gap-3">
{MAPPING_COLUMNS.map((column) => (
<Field key={column}>
<FieldLabel htmlFor={`${mappingId}-${column}`}>{labels[column]}</FieldLabel>
<NativeSelect
id={`${mappingId}-${column}`}
size="sm"
className="w-full"
value={draft.columns[column] ?? ""}
onChange={(event) =>
setDraft({
...draft,
columns: {
...draft.columns,
[column]: event.target.value || undefined,
},
})
}
>
<NativeSelectOption value="">–</NativeSelectOption>
{headers.map((header) => (
<NativeSelectOption key={header} value={header}>
{header}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
))}
<Field>
<FieldLabel htmlFor={`${mappingId}-key`}>
<Trans>Publication is</Trans>
</FieldLabel>
<NativeSelect
id={`${mappingId}-key`}
size="sm"
className="w-full"
value={draft.publicationKey}
onChange={(event) =>
setDraft({
...draft,
publicationKey: event.target.value as MetricCsvMapping["publicationKey"],
})
}
>
{(Object.keys(keyLabels) as MetricCsvMapping["publicationKey"][]).map(
(key) => (
<NativeSelectOption key={key} value={key}>
{keyLabels[key]}
</NativeSelectOption>
),
)}
</NativeSelect>
</Field>
<Field>
<FieldLabel htmlFor={`${mappingId}-unit`}>
<Trans>Revenue in</Trans>
</FieldLabel>
<div className="flex gap-2">
<NativeSelect
id={`${mappingId}-unit`}
size="sm"
className="w-full"
value={draft.revenueUnit}
onChange={(event) =>
setDraft({
...draft,
revenueUnit: event.target.value as MetricCsvMapping["revenueUnit"],
})
}
>
<NativeSelectOption value="major">1.00</NativeSelectOption>
<NativeSelectOption value="cents">100</NativeSelectOption>
</NativeSelect>
<Input
aria-label={t`Currency`}
className="h-7! w-16 uppercase"
maxLength={3}
value={draft.defaultCurrency}
onChange={(event) =>
setDraft({ ...draft, defaultCurrency: event.target.value.toUpperCase() })
}
/>
</div>
</Field>
</div>
</>
) : null}
{error ? (
<p role="alert" className="text-ui-body-sm text-destructive">
{error}
</p>
) : null}
<DialogFooter>
<Button type="submit" disabled={busy || !mapping || !csv}>
{fileName ? <Trans>Import {fileName}</Trans> : <Trans>Import</Trans>}
</Button>
</DialogFooter>
</form>
)}
</DialogContent>
</Dialog>
);
}

View File

@ -0,0 +1,61 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { FunnelTotals } from "../../../lib/media-library";
import { formatMoney, formatRate, funnelWidths } from "../../../lib/media-library";
/** Views → Clicks → Purchases → Revenue, each bar relative to views. */
export function Funnel({ totals }: { totals: FunnelTotals }) {
const { t, i18n } = useLingui();
const locale = i18n.locale;
const widths = funnelWidths(totals);
const number = (value: number) => new Intl.NumberFormat(locale).format(value);
const steps = [
{ key: "views", label: t`Views`, value: totals.views, width: widths.views, rate: null },
{
key: "clicks",
label: t`Clicks`,
value: totals.clicks,
width: widths.clicks,
rate: formatRate(totals.clicks, totals.views, locale),
},
{
key: "conversions",
label: t`Purchases`,
value: totals.conversions,
width: widths.conversions,
rate: formatRate(totals.conversions, totals.clicks, locale),
},
];
return (
<section
data-testid="media-funnel"
aria-label={t`Funnel`}
className="flex flex-col gap-2 rounded-xl border border-border bg-card p-4"
>
{steps.map((step) => (
<div key={step.key} className="grid grid-cols-[6.5rem_1fr_auto] items-center gap-3">
<span className="text-ui-body-sm text-muted-foreground">{step.label}</span>
<div className="h-6 overflow-hidden rounded-md bg-muted">
<div
className="h-full rounded-md bg-primary"
style={{ width: `${Math.max(step.value > 0 ? 1 : 0, step.width * 100)}%` }}
/>
</div>
<span className="min-w-24 text-right font-mono text-ui-body-sm tabular-nums">
{number(step.value)}
{step.rate ? <span className="ml-2 text-muted-foreground">{step.rate}</span> : null}
</span>
</div>
))}
<div className="grid grid-cols-[6.5rem_1fr] items-center gap-3">
<span className="text-ui-body-sm text-muted-foreground">
<Trans>Revenue</Trans>
</span>
<span className="text-right font-mono text-ui-body tabular-nums font-semibold">
{totals.revenue.length === 0
? "–"
: totals.revenue.map((row) => formatMoney(row.cents, row.currency, locale)).join(" · ")}
</span>
</div>
</section>
);
}

View File

@ -0,0 +1,172 @@
import { Plural, Trans, useLingui } from "@lingui/react/macro";
import type { MediaCollection } from "@rakazo/contracts";
import {
InputGroup,
InputGroupAddon,
InputGroupInput,
Tabs,
TabsList,
TabsTrigger,
Toggle,
} from "@rakazo/ui-web";
import { Search } from "lucide-react";
import type { LibraryFilter } from "../../../lib/media-library";
import { formatDuration, LIBRARY_FILTERS } from "../../../lib/media-library";
import { LoadingState } from "../../ai/primitives";
import { AssetThumb } from "./AssetTile";
export type LibraryMode = "sessions" | "themes" | "people";
/** Sessions | Themes | People, shared by the library and people views. */
export function LibraryModeToggle({
mode,
onChange,
}: {
mode: LibraryMode;
onChange: (mode: LibraryMode) => void;
}) {
const { t } = useLingui();
return (
<Tabs value={mode} onValueChange={(value) => onChange(value as LibraryMode)}>
<TabsList aria-label={t`Group by`}>
<TabsTrigger value="sessions">
<Trans>Sessions</Trans>
</TabsTrigger>
<TabsTrigger value="themes">
<Trans>Themes</Trans>
</TabsTrigger>
<TabsTrigger value="people">
<Trans>People</Trans>
</TabsTrigger>
</TabsList>
</Tabs>
);
}
export function LibrarySearch({
value,
onChange,
}: {
value: string;
onChange: (value: string) => void;
}) {
const { t } = useLingui();
return (
<InputGroup className="w-full max-w-64">
<InputGroupAddon>
<Search strokeWidth={1.8} aria-hidden="true" className="size-icon-md" />
</InputGroupAddon>
<InputGroupInput
type="search"
value={value}
onChange={(event) => onChange(event.target.value)}
placeholder={t`Search`}
aria-label={t`Search media`}
autoComplete="off"
/>
</InputGroup>
);
}
export function FilterChips({
value,
onChange,
}: {
value: ReadonlySet<LibraryFilter>;
onChange: (value: ReadonlySet<LibraryFilter>) => void;
}) {
const { t } = useLingui();
const labels: Record<LibraryFilter, string> = {
videos: t`Videos`,
photos: t`Photos`,
favorites: t`Favorites`,
best: t`Best`,
};
return (
<fieldset aria-label={t`Filters`} className="flex flex-wrap items-center gap-1.5">
{LIBRARY_FILTERS.map((filter) => (
<Toggle
key={filter}
size="sm"
variant="outline"
className="rounded-full"
pressed={value.has(filter)}
onPressedChange={(pressed) => {
const next = new Set(value);
if (pressed) next.add(filter);
else next.delete(filter);
onChange(next);
}}
>
{labels[filter]}
</Toggle>
))}
</fieldset>
);
}
/** Only rendered while indexing runs; disappears when the work is done. */
export function IndexingProgress({ done, total }: { done?: number; total?: number }) {
const { t } = useLingui();
return (
<div data-testid="media-indexing" className="flex items-center gap-3 px-4 py-2">
<LoadingState label={t`Indexing`} />
{total ? (
<span className="font-mono text-ui-body-sm tabular-nums text-muted-foreground">
{done ?? 0}/{total}
</span>
) : null}
</div>
);
}
export function CollectionCard({
collection,
onOpen,
}: {
collection: MediaCollection;
onOpen: () => void;
}) {
const range = collectionDateRange(collection);
const count = collection.assetCount;
return (
<button
type="button"
data-testid="media-collection-card"
onClick={onOpen}
className="group/card flex size-full min-w-0 flex-col gap-1.5 rounded-lg text-left outline-none focus-visible:ring-2 focus-visible:ring-ring"
>
<div className="relative min-h-0 flex-1 overflow-hidden rounded-lg bg-muted">
<AssetThumb
assetId={collection.coverAssetId}
hasThumbnail={collection.coverAssetId !== null}
className="transition-transform group-hover/card:scale-[1.02]"
/>
{collection.totalDurationMs > 0 ? (
<span className="absolute bottom-1.5 right-1.5 rounded bg-foreground/70 px-1 font-mono text-ui-caption tabular-nums text-background">
{formatDuration(collection.totalDurationMs)}
</span>
) : null}
</div>
<div className="min-w-0 px-0.5">
<div className="truncate text-ui-body-sm font-medium text-foreground" dir="auto">
{collection.title}
</div>
<div className="truncate text-ui-caption text-muted-foreground">
{range ? `${range} · ` : ""}
<Plural value={count} one="# item" other="# items" />
</div>
</div>
</button>
);
}
export function collectionDateRange(
collection: Pick<MediaCollection, "startAt" | "endAt">,
): string {
if (!collection.startAt) return "";
const start = new Date(collection.startAt);
const end = collection.endAt ? new Date(collection.endAt) : start;
const day = (date: Date) => date.toLocaleDateString(undefined, { dateStyle: "medium" });
return day(start) === day(end) ? day(start) : `${day(start)} – ${day(end)}`;
}

View File

@ -0,0 +1,184 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { MediaPublication } from "@rakazo/contracts";
import {
Button,
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
Field,
FieldLabel,
Input,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { useEffect, useId, useState } from "react";
import { dayToIso, todayDay } from "../../../lib/media-library";
import { rpcErrorMessage } from "../../../lib/media-studio";
import { rpc } from "../../../lib/rpc";
type Counts = { views: string; clicks: string; conversions: string; revenue: string };
const ZERO: Counts = { views: "0", clicks: "0", conversions: "0", revenue: "0" };
function count(value: string): number | null {
const parsed = Number(value);
return Number.isInteger(parsed) && parsed >= 0 ? parsed : null;
}
/** Manual entry for platforms without an export. */
export function MetricRecordDialog({
open,
onOpenChange,
publications,
onRecorded,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
publications: readonly MediaPublication[];
onRecorded: () => void;
}) {
const { t } = useLingui();
const ids = { publication: useId(), day: useId(), currency: useId() };
const [publicationId, setPublicationId] = useState("");
const [day, setDay] = useState(todayDay());
const [counts, setCounts] = useState<Counts>(ZERO);
const [currency, setCurrency] = useState("EUR");
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
if (!open) return;
setPublicationId(publications[0]?.id ?? "");
setDay(todayDay());
setCounts(ZERO);
setError(null);
}, [open, publications]);
const revenue = Number(counts.revenue.replace(",", "."));
const values = {
views: count(counts.views),
clicks: count(counts.clicks),
conversions: count(counts.conversions),
};
const observedAt = dayToIso(day);
const valid =
publicationId !== "" &&
observedAt !== null &&
values.views !== null &&
values.clicks !== null &&
values.conversions !== null &&
Number.isFinite(revenue) &&
revenue >= 0 &&
/^[A-Z]{3}$/.test(currency);
const save = async () => {
if (!valid || !observedAt) return;
setBusy(true);
setError(null);
try {
await rpc.media.metrics.record({
publicationId,
observedAt,
views: values.views ?? 0,
clicks: values.clicks ?? 0,
conversions: values.conversions ?? 0,
revenueCents: Math.round(revenue * 100),
currency,
});
onOpenChange(false);
onRecorded();
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not save.`));
} finally {
setBusy(false);
}
};
const numberField = (key: keyof Counts, label: string) => (
<Field key={key}>
<FieldLabel htmlFor={`${ids.publication}-${key}`}>{label}</FieldLabel>
<Input
id={`${ids.publication}-${key}`}
inputMode={key === "revenue" ? "decimal" : "numeric"}
value={counts[key]}
onChange={(event) => setCounts({ ...counts, [key]: event.target.value })}
/>
</Field>
);
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent data-testid="media-metric-record" className="sm:max-w-sm">
<DialogHeader>
<DialogTitle>
<Trans>Enter results</Trans>
</DialogTitle>
</DialogHeader>
<form
className="flex flex-col gap-3"
onSubmit={(event) => {
event.preventDefault();
void save();
}}
>
<Field>
<FieldLabel htmlFor={ids.publication}>
<Trans>Publication</Trans>
</FieldLabel>
<NativeSelect
id={ids.publication}
className="w-full"
value={publicationId}
onChange={(event) => setPublicationId(event.target.value)}
>
{publications.map((publication) => (
<NativeSelectOption key={publication.id} value={publication.id}>
{publication.title ?? publication.profileId}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
<Field>
<FieldLabel htmlFor={ids.day}>
<Trans>Date</Trans>
</FieldLabel>
<Input
id={ids.day}
type="date"
max={todayDay()}
value={day}
onChange={(event) => setDay(event.target.value)}
/>
</Field>
<div className="grid grid-cols-2 gap-3">
{numberField("views", t`Views`)}
{numberField("clicks", t`Clicks`)}
{numberField("conversions", t`Purchases`)}
<div className="flex items-end gap-2">
{numberField("revenue", t`Revenue`)}
<Input
id={ids.currency}
aria-label={t`Currency`}
className="w-16 uppercase"
maxLength={3}
value={currency}
onChange={(event) => setCurrency(event.target.value.toUpperCase())}
/>
</div>
</div>
{error ? (
<p role="alert" className="text-ui-body-sm text-destructive">
{error}
</p>
) : null}
<DialogFooter>
<Button type="submit" disabled={busy || !valid}>
<Trans>Save</Trans>
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
);
}

View File

@ -0,0 +1,199 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { AnonymizeMode, MediaPerson } from "@rakazo/contracts";
import { AnonymizeModeSchema } from "@rakazo/contracts";
import {
Badge,
Button,
Checkbox,
cn,
DropdownMenu,
DropdownMenuCheckboxItem,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
Input,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { Check, MoreHorizontal, ShieldAlert, ShieldCheck, UserRound } from "lucide-react";
import { useState } from "react";
import { consentState } from "../../../lib/media-library";
import { AssetThumb } from "./AssetTile";
export function useAnonymizeLabels(): Record<AnonymizeMode, string> {
const { t } = useLingui();
return {
none: t`Visible`,
blur: t`Blur`,
pixelate: t`Pixelate`,
solid: t`Solid color`,
sticker: t`Sticker`,
partial_eyes: t`Eye bar`,
partial_upper: t`Upper face`,
partial_lower: t`Lower face`,
reframe_out: t`Crop out`,
synthetic_identity: t`Synthetic face (experimental)`,
};
}
export function PersonCard({
person,
selected,
onSelect,
syntheticAvailable,
onUpdate,
onConsent,
onRemove,
}: {
person: MediaPerson;
selected: boolean;
onSelect: (selected: boolean) => void;
syntheticAvailable: boolean;
onUpdate: (patch: {
displayName?: string;
isSelf?: boolean;
confirmed?: boolean;
anonymizeByDefault?: AnonymizeMode;
}) => void;
onConsent: () => void;
onRemove: () => void;
}) {
const { t } = useLingui();
const modes = useAnonymizeLabels();
const [name, setName] = useState<string | null>(null);
const state = consentState(person);
const commitName = () => {
const trimmed = name?.trim();
setName(null);
if (trimmed && trimmed !== person.displayName) onUpdate({ displayName: trimmed });
};
return (
<div
data-testid="media-person"
data-person-id={person.id}
className={cn(
"flex flex-col gap-2.5 rounded-xl border border-border bg-card p-3",
selected && "ring-2 ring-primary",
)}
>
<div className="flex items-center gap-3">
<div className="relative size-12 shrink-0 overflow-hidden rounded-full bg-muted">
{person.coverThumbAssetId ? (
<AssetThumb assetId={person.coverThumbAssetId} hasThumbnail />
) : (
<UserRound
aria-hidden="true"
strokeWidth={1.6}
className="m-3 size-6 text-muted-foreground/60"
/>
)}
</div>
<div className="min-w-0 flex-1">
{name !== null ? (
<Input
autoFocus
aria-label={t`Name`}
value={name}
maxLength={60}
onChange={(event) => setName(event.target.value)}
onBlur={commitName}
onKeyDown={(event) => {
if (event.key === "Enter") commitName();
if (event.key === "Escape") setName(null);
}}
/>
) : (
<button
type="button"
className="max-w-full truncate text-left text-ui-body font-medium hover:underline"
onClick={() => setName(person.displayName)}
dir="auto"
>
{person.displayName}
</button>
)}
<div className="flex items-center gap-1.5 text-ui-caption text-muted-foreground">
{person.isSelf ? (
<Badge variant="secondary">
<Trans>You</Trans>
</Badge>
) : null}
<Trans>{person.faceTrackCount} appearances</Trans>
</div>
</div>
<Checkbox
aria-label={t`Select ${person.displayName}`}
checked={selected}
onCheckedChange={(checked) => onSelect(checked === true)}
/>
<DropdownMenu>
<DropdownMenuTrigger
render={<Button variant="ghost" size="icon-sm" aria-label={t`More`} />}
>
<MoreHorizontal />
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuCheckboxItem
checked={person.isSelf}
onCheckedChange={(checked) => onUpdate({ isSelf: checked === true })}
>
<Trans>This is me</Trans>
</DropdownMenuCheckboxItem>
<DropdownMenuSeparator />
<DropdownMenuItem variant="destructive" onClick={onRemove}>
<Trans>Delete face data</Trans>
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
<NativeSelect
size="sm"
className="w-full"
aria-label={t`Anonymization`}
value={person.anonymizeByDefault}
onChange={(event) => onUpdate({ anonymizeByDefault: event.target.value as AnonymizeMode })}
>
{AnonymizeModeSchema.options.map((mode) => (
<NativeSelectOption
key={mode}
value={mode}
disabled={mode === "synthetic_identity" && !syntheticAvailable}
>
{modes[mode]}
</NativeSelectOption>
))}
</NativeSelect>
<div className="flex items-center gap-2">
{person.confirmed ? (
<span className="flex items-center gap-1 text-ui-body-sm text-muted-foreground">
<Check aria-hidden="true" className="size-icon-sm" />
<Trans>Confirmed</Trans>
</span>
) : (
<Button size="sm" variant="outline" onClick={() => onUpdate({ confirmed: true })}>
<Trans>Confirm</Trans>
</Button>
)}
<Button
size="sm"
variant={state === "complete" ? "ghost" : "outline"}
className="ml-auto"
data-consent={state}
onClick={onConsent}
>
{state === "complete" ? (
<ShieldCheck aria-hidden="true" className="text-success" />
) : (
<ShieldAlert aria-hidden="true" className="text-warning" />
)}
{state === "missing" ? <Trans>Add consent</Trans> : <Trans>Consent</Trans>}
</Button>
</div>
</div>
);
}

View File

@ -0,0 +1,128 @@
import { cn } from "@rakazo/ui-web";
import { useVirtualizer } from "@tanstack/react-virtual";
import type { ReactNode } from "react";
import { useEffect, useLayoutEffect, useRef, useState } from "react";
import { gridColumns, gridRowCount } from "../../../lib/media-library";
/**
* Row-virtualized grid: only the rows in view (plus overscan) are mounted, so
* 50k items cost the same DOM as fifty. Tiles share one size derived from the
* container width; `onNearEnd` asks for the next page before the end shows.
*/
export function VirtualGrid({
count,
renderItem,
getKey,
minTileWidth = 168,
gap = 8,
tileAspect = 9 / 16,
extraHeight = 0,
overscan = 4,
onNearEnd,
footer,
className,
label,
testId,
}: {
count: number;
renderItem: (index: number) => ReactNode;
getKey?: (index: number) => string | number;
minTileWidth?: number;
gap?: number;
/** Media height / width. */
tileAspect?: number;
/** Fixed height under the media (title lines). */
extraHeight?: number;
overscan?: number;
onNearEnd?: () => void;
footer?: ReactNode;
className?: string;
label?: string;
testId?: string;
}) {
const scrollRef = useRef<HTMLElement>(null);
const [width, setWidth] = useState(0);
useLayoutEffect(() => {
const element = scrollRef.current;
if (!element) return;
const measure = () => setWidth(element.clientWidth);
measure();
const observer = new ResizeObserver(measure);
observer.observe(element);
return () => observer.disconnect();
}, []);
const padding = 16;
const inner = Math.max(0, width - padding * 2);
const columns = gridColumns(inner, minTileWidth, gap);
const tileWidth = columns > 0 ? (inner - gap * (columns - 1)) / columns : minTileWidth;
const rowHeight = Math.round(tileWidth * tileAspect) + extraHeight + gap;
const rows = gridRowCount(count, columns);
const virtualizer = useVirtualizer({
count: rows,
getScrollElement: () => scrollRef.current,
estimateSize: () => rowHeight,
overscan,
paddingStart: padding,
paddingEnd: padding,
});
useEffect(() => {
virtualizer.measure();
}, [virtualizer, rowHeight]);
const items = virtualizer.getVirtualItems();
const lastRow = items.at(-1)?.index ?? -1;
const nearEnd = rows > 0 && lastRow >= rows - overscan - 1;
const latestNearEnd = useRef(onNearEnd);
latestNearEnd.current = onNearEnd;
useEffect(() => {
if (nearEnd) latestNearEnd.current?.();
}, [nearEnd, count]);
return (
<section
ref={scrollRef}
aria-label={label}
data-testid={testId}
data-count={count}
className={cn("rk-scroll relative min-h-0 flex-1 overflow-y-auto", className)}
>
<div style={{ height: virtualizer.getTotalSize(), position: "relative" }}>
{width > 0
? items.map((row) => {
const start = row.index * columns;
const end = Math.min(count, start + columns);
return (
<div
key={row.key}
className="absolute inset-x-4 grid"
style={{
top: row.start,
gap,
gridTemplateColumns: `repeat(${columns}, minmax(0, 1fr))`,
}}
>
{Array.from({ length: end - start }, (_, offset) => {
const index = start + offset;
return (
<div
key={getKey ? getKey(index) : index}
style={{ height: rowHeight - gap }}
className="min-w-0"
>
{renderItem(index)}
</div>
);
})}
</div>
);
})
: null}
</div>
{footer}
</section>
);
}

View File

@ -0,0 +1,72 @@
import type { PhotosSourceStatus, PhotosSyncOutcome, PhotosSyncProgress } from "@rakazo/contracts";
import { useCallback, useEffect, useState } from "react";
import { desktopBridge } from "../../../lib/desktop";
import { selectedSpaceId } from "../../../lib/rpc";
/**
* Apple Photos runs in the desktop main process; the renderer only asks for a
* sync and shows progress. Absent outside the macOS desktop app.
*/
export function usePhotosSync(onFinished?: (outcome: PhotosSyncOutcome) => void) {
const photos = desktopBridge()?.media?.photos;
const [status, setStatus] = useState<PhotosSourceStatus>({ available: false });
const [progress, setProgress] = useState<PhotosSyncProgress | null>(null);
const [syncing, setSyncing] = useState(false);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
if (!photos) return;
let active = true;
photos
.status()
.then((value) => {
if (!active) return;
setStatus(value);
if (value.available && value.syncing) setSyncing(true);
})
.catch(() => undefined);
const off = photos.onProgress((value) => {
if (active) setProgress(value);
});
return () => {
active = false;
off();
};
}, [photos]);
const sync = useCallback(async () => {
if (!photos) return;
setSyncing(true);
setError(null);
try {
const outcome = await photos.sync({ spaceId: selectedSpaceId() });
if (!outcome.ok) setError(outcome.message);
onFinished?.(outcome);
} catch (cause) {
setError(cause instanceof Error ? cause.message : String(cause));
} finally {
setSyncing(false);
setProgress(null);
photos
.status()
.then(setStatus)
.catch(() => undefined);
}
}, [photos, onFinished]);
const cancel = useCallback(() => {
void photos?.cancel().catch(() => undefined);
}, [photos]);
return {
available: Boolean(photos) && status.available,
denied:
status.available &&
(status.authorization === "denied" || status.authorization === "restricted"),
syncing,
progress,
error,
sync,
cancel,
};
}

View File

@ -0,0 +1,91 @@
import { describe, expect, it, vi } from "vitest";
import {
createMediaObjectUrlCache,
mediaAssetPath,
mediaBundlePath,
mediaRenderFilePath,
mediaUploadChunkPath,
} from "./media-bytes";
function setup(idleLimit = 2) {
let next = 0;
const revoked: string[] = [];
const loads: string[] = [];
const pending = new Map<string, (blob: Blob) => void>();
const cache = createMediaObjectUrlCache({
load: (path, signal) => {
loads.push(path);
return new Promise<Blob>((resolve, reject) => {
pending.set(path, resolve);
signal.addEventListener("abort", () => reject(new DOMException("Aborted", "AbortError")));
});
},
createUrl: () => {
next += 1;
return `blob:${next}`;
},
revokeUrl: (url) => revoked.push(url),
spaceId: () => "space-1",
idleLimit,
});
const finish = (path: string) => pending.get(path)?.(new Blob(["x"]));
return { cache, revoked, loads, finish };
}
describe("media byte paths", () => {
it("encodes ids into the byte routes", () => {
expect(mediaAssetPath("a/b", "thumb")).toBe("/api/media/assets/a%2Fb/thumb");
expect(mediaRenderFilePath("r1", "video.mp4", { download: true })).toBe(
"/api/media/renders/r1/files/video.mp4?download=1",
);
expect(mediaBundlePath("p1")).toBe("/api/media/bundles/p1");
expect(mediaUploadChunkPath("u1", 3)).toBe("/api/media/uploads/u1/chunks/3");
});
});
describe("media object URL cache", () => {
it("shares one load between holders", async () => {
const { cache, loads, finish } = setup();
const a = cache.acquire("/t/1", { keep: true });
const b = cache.acquire("/t/1", { keep: true });
finish("/t/1");
await expect(a.promise).resolves.toBe("blob:1");
await expect(b.promise).resolves.toBe("blob:1");
expect(loads).toEqual(["/t/1"]);
});
it("keeps thumbnails idle up to the limit, evicting the oldest", async () => {
const { cache, revoked, finish } = setup(2);
for (const id of ["1", "2", "3"]) {
const handle = cache.acquire(`/t/${id}`, { keep: true });
finish(`/t/${id}`);
await handle.promise;
handle.release();
}
expect(cache.size()).toBe(2);
expect(revoked).toEqual(["blob:1"]);
});
it("drops large files as soon as the last holder releases", async () => {
const { cache, revoked, finish } = setup();
const handle = cache.acquire("/p/1");
finish("/p/1");
await handle.promise;
handle.release();
handle.release();
expect(cache.size()).toBe(0);
expect(revoked).toEqual(["blob:1"]);
});
it("aborts a load nobody waits for anymore", async () => {
const { cache } = setup();
const handle = cache.acquire("/p/2");
const settled = vi.fn();
handle.promise.catch(settled);
handle.release();
await Promise.resolve();
await Promise.resolve();
expect(settled).toHaveBeenCalled();
expect(cache.size()).toBe(0);
});
});

View File

@ -0,0 +1,218 @@
import { useEffect, useState } from "react";
import { selectedSpaceId, withSpaceHeaders } from "./rpc";
/*
* Media byte routes (thumbnails, proxies, render files, bundles, upload chunks).
* They need the session cookie and the space header, which an `<img src>` or
* `<video src>` cannot send, so bytes are fetched here and handed to elements
* as object URLs. Thumbnails stay cached after release; large files do not.
*/
export type MediaAssetFile = "thumb" | "proxy";
export function mediaAssetPath(assetId: string, file: MediaAssetFile): string {
return `/api/media/assets/${encodeURIComponent(assetId)}/${file}`;
}
export function mediaRenderFilePath(
renderId: string,
fileName: string,
options: { download?: boolean } = {},
): string {
const base = `/api/media/renders/${encodeURIComponent(renderId)}/files/${encodeURIComponent(fileName)}`;
return options.download ? `${base}?download=1` : base;
}
export function mediaBundlePath(publicationId: string): string {
return `/api/media/bundles/${encodeURIComponent(publicationId)}`;
}
export function mediaUploadChunkPath(uploadId: string, index: number): string {
return `/api/media/uploads/${encodeURIComponent(uploadId)}/chunks/${index}`;
}
export class MediaFetchError extends Error {
constructor(readonly status: number) {
super(`Media request failed (${status})`);
this.name = "MediaFetchError";
}
}
/** Fetch a media byte route with the session cookie and the selected space. */
export async function fetchMedia(path: string, init: RequestInit = {}): Promise<Response> {
const response = await fetch(path, {
...init,
credentials: "include",
headers: withSpaceHeaders(init.headers),
});
if (!response.ok) throw new MediaFetchError(response.status);
return response;
}
/** Save a media file through the browser's download flow. */
export async function downloadMedia(path: string, fileName: string): Promise<void> {
const blob = await (await fetchMedia(path)).blob();
const url = URL.createObjectURL(blob);
try {
const anchor = document.createElement("a");
anchor.href = url;
anchor.download = fileName;
anchor.click();
} finally {
setTimeout(() => URL.revokeObjectURL(url), 30_000);
}
}
type Entry = {
promise: Promise<string>;
url: string | null;
refs: number;
/** Keep the object URL after the last release (small files such as thumbnails). */
keep: boolean;
controller: AbortController;
};
/** Idle thumbnails kept around for scrolling back; each one is a few KB. */
export const MEDIA_IDLE_CACHE_LIMIT = 400;
export interface MediaObjectUrlCache {
acquire(
path: string,
options?: { keep?: boolean },
): { promise: Promise<string>; release: () => void };
/** Number of entries, including idle ones (for tests). */
size(): number;
clear(): void;
}
export function createMediaObjectUrlCache(deps: {
load: (path: string, signal: AbortSignal) => Promise<Blob>;
createUrl: (blob: Blob) => string;
revokeUrl: (url: string) => void;
spaceId: () => string | null;
idleLimit?: number;
}): MediaObjectUrlCache {
const entries = new Map<string, Entry>();
const idleLimit = deps.idleLimit ?? MEDIA_IDLE_CACHE_LIMIT;
const drop = (key: string, entry: Entry) => {
entries.delete(key);
entry.controller.abort();
if (entry.url) deps.revokeUrl(entry.url);
};
const trim = () => {
let idle = 0;
for (const entry of entries.values()) if (entry.refs === 0) idle += 1;
for (const [key, entry] of entries) {
if (idle <= idleLimit) break;
if (entry.refs > 0) continue;
drop(key, entry);
idle -= 1;
}
};
return {
acquire(path, options = {}) {
const key = `${deps.spaceId() ?? ""}|${path}`;
let entry = entries.get(key);
if (entry) {
// Re-insert so the map order stays least recently used first.
entries.delete(key);
entries.set(key, entry);
} else {
const controller = new AbortController();
const created: Entry = {
url: null,
refs: 0,
keep: options.keep ?? false,
controller,
promise: Promise.resolve(""),
};
created.promise = deps.load(path, controller.signal).then(
(blob) => {
const url = deps.createUrl(blob);
if (entries.get(key) !== created) {
deps.revokeUrl(url);
throw new DOMException("Released", "AbortError");
}
created.url = url;
return url;
},
(error: unknown) => {
if (entries.get(key) === created) entries.delete(key);
throw error;
},
);
// Callers that went away never observe the rejection.
created.promise.catch(() => undefined);
entries.set(key, created);
entry = created;
}
const held = entry;
held.refs += 1;
let released = false;
return {
promise: held.promise,
release: () => {
if (released) return;
released = true;
held.refs -= 1;
if (held.refs > 0 || entries.get(key) !== held) return;
if (held.keep && held.url) trim();
else drop(key, held);
},
};
},
size: () => entries.size,
clear: () => {
for (const [key, entry] of entries) drop(key, entry);
},
};
}
const sharedCache = createMediaObjectUrlCache({
load: async (path, signal) => (await fetchMedia(path, { signal })).blob(),
createUrl: (blob) => URL.createObjectURL(blob),
revokeUrl: (url) => URL.revokeObjectURL(url),
spaceId: selectedSpaceId,
});
export type MediaObjectUrlState =
| { status: "idle" | "loading"; url: null }
| { status: "ready"; url: string }
| { status: "missing"; url: null };
/**
* Object URL for a media byte route, or null while loading, when `path` is null,
* or when the file does not exist (yet). `keep` caches the bytes after unmount.
*/
export function useMediaObjectUrl(
path: string | null,
options: { keep?: boolean } = {},
): MediaObjectUrlState {
const keep = options.keep ?? false;
const [state, setState] = useState<MediaObjectUrlState>({ status: "idle", url: null });
useEffect(() => {
if (!path) {
setState({ status: "idle", url: null });
return;
}
let active = true;
setState({ status: "loading", url: null });
const handle = sharedCache.acquire(path, { keep });
handle.promise.then(
(url) => {
if (active) setState({ status: "ready", url });
},
() => {
if (active) setState({ status: "missing", url: null });
},
);
return () => {
active = false;
handle.release();
};
}, [path, keep]);
return state;
}

View File

@ -0,0 +1,182 @@
import { describe, expect, it } from "vitest";
import {
assetQueryFor,
BEST_QUALITY,
consentInput,
consentState,
csvHeaders,
dayToIso,
formatDuration,
funnelTotals,
funnelWidths,
gridColumns,
gridRowCount,
guessColumns,
guessDelimiter,
guessPublicationKey,
hasAssetQuery,
type LibraryFilter,
scrubFraction,
splitCsvLine,
} from "./media-library";
describe("library filters", () => {
it("maps chips to an asset query", () => {
const filters = new Set<LibraryFilter>(["videos", "favorites", "best"]);
expect(assetQueryFor(filters, " beach ")).toEqual({
kinds: ["video"],
favoritesOnly: true,
minQuality: BEST_QUALITY,
text: "beach",
});
expect(assetQueryFor(new Set(["photos"]), "")).toEqual({ kinds: ["photo", "live_photo"] });
expect(assetQueryFor(new Set(), "")).toEqual({});
});
it("knows when a query is active", () => {
expect(hasAssetQuery(new Set(), " ")).toBe(false);
expect(hasAssetQuery(new Set(), "x")).toBe(true);
expect(hasAssetQuery(new Set(["videos"]), "")).toBe(true);
});
});
describe("grid geometry", () => {
it("fits columns to the width", () => {
expect(gridColumns(0, 160, 8)).toBe(1);
expect(gridColumns(159, 160, 8)).toBe(1);
expect(gridColumns(328, 160, 8)).toBe(2);
expect(gridColumns(1000, 160, 8)).toBe(6);
});
it("counts rows for 50k assets without materializing them", () => {
expect(gridRowCount(50_000, 6)).toBe(8334);
expect(gridRowCount(0, 6)).toBe(0);
expect(gridRowCount(5, 0)).toBe(5);
});
it("clamps the scrub position", () => {
expect(scrubFraction(50, 0, 100)).toBe(0.5);
expect(scrubFraction(-10, 0, 100)).toBe(0);
expect(scrubFraction(200, 0, 100)).toBe(1);
expect(scrubFraction(5, 0, 0)).toBe(0);
});
});
describe("formatDuration", () => {
it("formats clip lengths", () => {
expect(formatDuration(4000)).toBe("0:04");
expect(formatDuration(62_400)).toBe("1:02");
expect(formatDuration(3_723_000)).toBe("1:02:03");
expect(formatDuration(null)).toBe("");
});
});
describe("consent", () => {
const now = new Date("2026-10-03T10:00:00Z");
it("records platform verification with its date", () => {
expect(
consentInput(
"p1",
{ consentDate: "2026-10-01", method: "platform_verified", verifiedDate: "2026-09-30" },
now,
),
).toEqual({
personId: "p1",
consentArtifactId: null,
consentRecordedAt: "2026-10-01T12:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-30T12:00:00.000Z",
});
});
it("drops the verification date when there is no age proof", () => {
expect(
consentInput("p1", { consentDate: "2026-10-01", method: "none", verifiedDate: "x" }, now)
?.ageVerifiedAt,
).toBeNull();
});
it("rejects incomplete or future dates", () => {
expect(
consentInput("p1", { consentDate: "", method: "none", verifiedDate: "" }, now),
).toBeNull();
expect(
consentInput(
"p1",
{ consentDate: "2026-10-01", method: "document_on_file", verifiedDate: "" },
now,
),
).toBeNull();
expect(
consentInput("p1", { consentDate: "2026-12-01", method: "none", verifiedDate: "" }, now),
).toBeNull();
expect(dayToIso("2026-13-45")).toBeNull();
});
it("summarizes a person's state", () => {
const base = { consentRecordedAt: null, ageVerificationMethod: "none", ageVerifiedAt: null };
expect(consentState({ ...base, ageVerificationMethod: "none" })).toBe("missing");
expect(consentState({ ...base, consentRecordedAt: "2026-10-01T12:00:00.000Z" } as never)).toBe(
"consent_only",
);
expect(
consentState({
consentRecordedAt: "2026-10-01T12:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-10-01T12:00:00.000Z",
}),
).toBe("complete");
});
});
describe("csv mapping", () => {
it("splits quoted cells", () => {
expect(splitCsvLine('a,"b, c","d ""e"""', ",")).toEqual(["a", "b, c", 'd "e"']);
});
it("detects the delimiter and headers", () => {
const csv = "Datum;utm_content;Aufrufe;Klicks;Käufe;Umsatz\n2026-10-01;a;1;2;3;4";
const delimiter = guessDelimiter(csv.split("\n")[0] ?? "");
expect(delimiter).toBe(";");
const headers = csvHeaders(csv, delimiter);
expect(headers).toEqual(["Datum", "utm_content", "Aufrufe", "Klicks", "Käufe", "Umsatz"]);
expect(guessColumns(headers)).toEqual({
publication: "utm_content",
observedAt: "Datum",
views: "Aufrufe",
clicks: "Klicks",
conversions: "Käufe",
revenue: "Umsatz",
});
});
it("guesses how publications are identified", () => {
expect(guessPublicationKey("utm_content")).toBe("utm_content");
expect(guessPublicationKey("tracking_url")).toBe("tracking_url");
expect(guessPublicationKey("Post URL")).toBe("external_url");
expect(guessPublicationKey("publication_id")).toBe("id");
expect(guessPublicationKey(undefined)).toBe("utm_content");
});
});
describe("funnel", () => {
it("sums steps and keeps currencies apart", () => {
const totals = funnelTotals([
{ views: 1000, clicks: 100, conversions: 10, revenueCents: 9900, currency: "EUR" },
{ views: 500, clicks: 20, conversions: 1, revenueCents: 1500, currency: "USD" },
{ views: 10, clicks: 0, conversions: 0, revenueCents: 0, currency: "EUR" },
]);
expect(totals).toEqual({
views: 1510,
clicks: 120,
conversions: 11,
revenue: [
{ currency: "EUR", cents: 9900 },
{ currency: "USD", cents: 1500 },
],
});
expect(funnelWidths(totals).views).toBe(1);
expect(funnelWidths({ views: 0, clicks: 0, conversions: 0 }).views).toBe(0);
});
});

View File

@ -0,0 +1,315 @@
import type {
AgeVerificationMethod,
ConsentInput,
FunnelRow,
MediaAssetKind,
MediaAssetQuery,
MediaPerson,
MetricCsvMapping,
} from "@rakazo/contracts";
/* Pure helpers behind the Media Studio library, people and results views. */
// ---------------------------------------------------------------------------
// Library filters
// ---------------------------------------------------------------------------
export const LIBRARY_FILTERS = ["videos", "photos", "favorites", "best"] as const;
export type LibraryFilter = (typeof LIBRARY_FILTERS)[number];
/** Quality floor for the "best" chip. */
export const BEST_QUALITY = 0.6;
export function assetQueryFor(
filters: ReadonlySet<LibraryFilter>,
text: string,
): Partial<MediaAssetQuery> {
const query: Partial<MediaAssetQuery> = {};
const kinds: MediaAssetKind[] = [];
if (filters.has("videos")) kinds.push("video");
if (filters.has("photos")) kinds.push("photo", "live_photo");
if (kinds.length > 0) query.kinds = kinds;
if (filters.has("favorites")) query.favoritesOnly = true;
if (filters.has("best")) query.minQuality = BEST_QUALITY;
const trimmed = text.trim();
if (trimmed) query.text = trimmed.slice(0, 200);
return query;
}
export function hasAssetQuery(filters: ReadonlySet<LibraryFilter>, text: string): boolean {
return filters.size > 0 || text.trim().length > 0;
}
/**
* Ingest states that mean the server is working on an asset. "indexed" is left
* out: device-held assets wait there until the device uploads them.
*/
export const INGEST_PENDING_STATES = ["proxying", "proxied", "analyzing"] as const;
// ---------------------------------------------------------------------------
// Grid geometry
// ---------------------------------------------------------------------------
/** Columns that fit `width` with tiles at least `minTile` wide. */
export function gridColumns(width: number, minTile: number, gap: number): number {
if (!Number.isFinite(width) || width <= 0) return 1;
return Math.max(1, Math.floor((width + gap) / (minTile + gap)));
}
export function gridRowCount(items: number, columns: number): number {
return Math.ceil(items / Math.max(1, columns));
}
// ---------------------------------------------------------------------------
// Formatting
// ---------------------------------------------------------------------------
export function formatDuration(ms: number | null | undefined): string {
if (ms == null || !Number.isFinite(ms) || ms < 0) return "";
const total = Math.round(ms / 1000);
const hours = Math.floor(total / 3600);
const minutes = Math.floor((total % 3600) / 60);
const seconds = String(total % 60).padStart(2, "0");
if (hours > 0) return `${hours}:${String(minutes).padStart(2, "0")}:${seconds}`;
return `${minutes}:${seconds}`;
}
export function formatMoney(cents: number, currency: string, locale?: string): string {
try {
return new Intl.NumberFormat(locale, { style: "currency", currency }).format(cents / 100);
} catch {
return `${(cents / 100).toFixed(2)} ${currency}`;
}
}
export function formatRate(numerator: number, denominator: number, locale?: string): string {
if (denominator <= 0) return "–";
return new Intl.NumberFormat(locale, { style: "percent", maximumFractionDigits: 1 }).format(
numerator / denominator,
);
}
/** Position of the pointer across an element as 0..1, for hover scrubbing. */
export function scrubFraction(clientX: number, left: number, width: number): number {
if (width <= 0) return 0;
return Math.min(1, Math.max(0, (clientX - left) / width));
}
// ---------------------------------------------------------------------------
// Consent
// ---------------------------------------------------------------------------
export type ConsentState = "complete" | "consent_only" | "missing";
export function consentState(
person: Pick<MediaPerson, "consentRecordedAt" | "ageVerificationMethod" | "ageVerifiedAt">,
): ConsentState {
if (!person.consentRecordedAt) return "missing";
if (person.ageVerificationMethod !== "none" && person.ageVerifiedAt) return "complete";
return "consent_only";
}
export interface ConsentForm {
/** yyyy-mm-dd from a date input. */
consentDate: string;
method: AgeVerificationMethod;
/** yyyy-mm-dd; ignored when method is "none". */
verifiedDate: string;
}
const DAY = /^\d{4}-\d{2}-\d{2}$/;
/** yyyy-mm-dd → ISO timestamp at UTC noon, so the calendar day survives any time zone. */
export function dayToIso(day: string): string | null {
if (!DAY.test(day)) return null;
const date = new Date(`${day}T12:00:00.000Z`);
return Number.isNaN(date.getTime()) ? null : date.toISOString();
}
export function isoToDay(iso: string | null | undefined): string {
return iso ? iso.slice(0, 10) : "";
}
export function todayDay(now: Date = new Date()): string {
const local = new Date(now.getTime() - now.getTimezoneOffset() * 60_000);
return local.toISOString().slice(0, 10);
}
/** Build the recordConsent input, or null when the form is incomplete. */
export function consentInput(
personId: string,
form: ConsentForm,
now: Date = new Date(),
): ConsentInput | null {
const consentRecordedAt = dayToIso(form.consentDate);
if (!consentRecordedAt) return null;
const today = todayDay(now);
if (form.consentDate > today) return null;
if (form.method === "none") {
return {
personId,
consentArtifactId: null,
consentRecordedAt,
ageVerificationMethod: "none",
ageVerifiedAt: null,
};
}
const ageVerifiedAt = dayToIso(form.verifiedDate);
if (!ageVerifiedAt || form.verifiedDate > today) return null;
return {
personId,
consentArtifactId: null,
consentRecordedAt,
ageVerificationMethod: form.method,
ageVerifiedAt,
};
}
// ---------------------------------------------------------------------------
// CSV metric import
// ---------------------------------------------------------------------------
export type CsvDelimiter = MetricCsvMapping["delimiter"];
/** Split one CSV line, honoring double-quoted fields. */
export function splitCsvLine(line: string, delimiter: CsvDelimiter): string[] {
const cells: string[] = [];
let cell = "";
let quoted = false;
for (let index = 0; index < line.length; index += 1) {
const char = line[index];
if (quoted) {
if (char === '"' && line[index + 1] === '"') {
cell += '"';
index += 1;
} else if (char === '"') {
quoted = false;
} else {
cell += char;
}
} else if (char === '"') {
quoted = true;
} else if (char === delimiter) {
cells.push(cell.trim());
cell = "";
} else {
cell += char;
}
}
cells.push(cell.trim());
return cells;
}
export function firstCsvLine(csv: string): string {
const withoutBom = csv.startsWith("") ? csv.slice(1) : csv;
return withoutBom.split(/\r?\n/, 1)[0] ?? "";
}
export function guessDelimiter(headerLine: string): CsvDelimiter {
const counts: Array<[CsvDelimiter, number]> = [
[",", headerLine.split(",").length],
[";", headerLine.split(";").length],
["\t", headerLine.split("\t").length],
];
counts.sort((a, b) => b[1] - a[1]);
return counts[0]?.[0] ?? ",";
}
export function csvHeaders(csv: string, delimiter: CsvDelimiter): string[] {
return splitCsvLine(firstCsvLine(csv), delimiter).filter((header) => header.length > 0);
}
export type MappingColumn = keyof MetricCsvMapping["columns"];
export const MAPPING_COLUMNS: MappingColumn[] = [
"publication",
"observedAt",
"views",
"clicks",
"conversions",
"revenue",
"currency",
];
const COLUMN_HINTS: Record<MappingColumn, RegExp> = {
publication: /(utm.?content|tracking|publication|post|video|url|link|^id$)/i,
observedAt: /(date|day|time|observed|datum)/i,
views: /(views?|impressions?|plays?|aufrufe)/i,
clicks: /(clicks?|taps?|klicks?)/i,
conversions: /(conversions?|purchases?|sales|orders?|käufe|kaeufe)/i,
revenue: /(revenue|earnings|amount|umsatz|income)/i,
currency: /(currency|währung|waehrung)/i,
};
/** Best guess of which header feeds which metric; every header is used at most once. */
export function guessColumns(headers: string[]): Partial<Record<MappingColumn, string>> {
const used = new Set<string>();
const guess: Partial<Record<MappingColumn, string>> = {};
for (const column of MAPPING_COLUMNS) {
const match = headers.find((header) => !used.has(header) && COLUMN_HINTS[column].test(header));
if (match) {
guess[column] = match;
used.add(match);
}
}
return guess;
}
export function guessPublicationKey(
header: string | undefined,
): MetricCsvMapping["publicationKey"] {
if (!header) return "utm_content";
if (/utm/i.test(header)) return "utm_content";
if (/track/i.test(header)) return "tracking_url";
if (/(url|link)/i.test(header)) return "external_url";
if (/id$/i.test(header)) return "id";
return "utm_content";
}
// ---------------------------------------------------------------------------
// Funnel
// ---------------------------------------------------------------------------
export interface FunnelTotals {
views: number;
clicks: number;
conversions: number;
/** One amount per currency; revenue never mixes currencies. */
revenue: Array<{ currency: string; cents: number }>;
}
export function funnelTotals(
rows: ReadonlyArray<
Pick<FunnelRow, "views" | "clicks" | "conversions" | "revenueCents" | "currency">
>,
): FunnelTotals {
const revenue = new Map<string, number>();
let views = 0;
let clicks = 0;
let conversions = 0;
for (const row of rows) {
views += row.views;
clicks += row.clicks;
conversions += row.conversions;
if (row.revenueCents > 0 || !revenue.has(row.currency)) {
revenue.set(row.currency, (revenue.get(row.currency) ?? 0) + row.revenueCents);
}
}
return {
views,
clicks,
conversions,
revenue: [...revenue.entries()]
.map(([currency, cents]) => ({ currency, cents }))
.sort((a, b) => b.cents - a.cents || a.currency.localeCompare(b.currency)),
};
}
/** Bar widths for the funnel steps, each relative to views (0..1). */
export function funnelWidths(totals: Pick<FunnelTotals, "views" | "clicks" | "conversions">) {
const base = Math.max(totals.views, totals.clicks, totals.conversions, 1);
return {
views: totals.views / base,
clicks: totals.clicks / base,
conversions: totals.conversions / base,
};
}

View File

@ -0,0 +1,51 @@
import { useEffect, useRef } from "react";
/** Media has no realtime subscription: clients poll while work runs, then stop. */
export const MEDIA_POLL_MS = 2000;
/**
* Call `tick` every `intervalMs` while `active`. Ticks never overlap: the next
* one is scheduled after the previous settles. Hidden tabs skip ticks.
*/
export function usePolling(
tick: () => unknown,
active: boolean,
intervalMs: number = MEDIA_POLL_MS,
): void {
const latest = useRef(tick);
latest.current = tick;
useEffect(() => {
if (!active) return;
let stopped = false;
let timer: ReturnType<typeof setTimeout> | undefined;
const schedule = () => {
if (stopped) return;
timer = setTimeout(run, intervalMs);
};
const run = async () => {
if (stopped) return;
if (typeof document !== "undefined" && document.hidden) {
schedule();
return;
}
try {
await latest.current();
} catch {
// The caller renders its own error state; polling continues.
}
schedule();
};
schedule();
return () => {
stopped = true;
if (timer) clearTimeout(timer);
};
}, [active, intervalMs]);
}
/** Render states that still change on the server. */
export const MEDIA_RENDER_ACTIVE_STATES = ["queued", "rendering", "gating"] as const;
export function isRenderActive(status: string): boolean {
return (MEDIA_RENDER_ACTIVE_STATES as readonly string[]).includes(status);
}

View File

@ -0,0 +1,51 @@
import type { MediaCapabilities } from "@rakazo/contracts";
import { createContext, useContext, useEffect, useState } from "react";
import { rpc } from "./rpc";
/** What every Media Studio view may read about the deployment. */
export interface MediaStudioEnv {
capabilities: MediaCapabilities | null;
/** Folder sources live on the host, so only the deployment owner links them. */
canLinkFolder: boolean;
}
export const MediaStudioContext = createContext<MediaStudioEnv>({
capabilities: null,
canLinkFolder: false,
});
export function useMediaStudio(): MediaStudioEnv {
return useContext(MediaStudioContext);
}
/** Load media capabilities once; null until known, and on failure. */
export function useMediaCapabilities(enabled = true): MediaCapabilities | null {
const [capabilities, setCapabilities] = useState<MediaCapabilities | null>(null);
useEffect(() => {
if (!enabled) return;
let active = true;
rpc.media
.capabilities({})
.then((value) => {
if (active) setCapabilities(value);
})
.catch(() => undefined);
return () => {
active = false;
};
}, [enabled]);
return capabilities;
}
/** Media Studio entry points stay hidden unless the deployment configured it. */
export function useMediaStudioAvailable(enabled = true): boolean {
return useMediaCapabilities(enabled)?.configured === true;
}
export function rpcErrorMessage(error: unknown, fallback: string): string {
if (error && typeof error === "object" && "message" in error) {
const message = String((error as { message: unknown }).message);
if (message && !/^(Internal|Bad Request|Precondition)/i.test(message)) return message;
}
return fallback;
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-editor slice replaces this view. */
export function BrandKitView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,147 @@
import { Plural, Trans, useLingui } from "@lingui/react/macro";
import type { MediaAsset, MediaCollection } from "@rakazo/contracts";
import { Button, Input } from "@rakazo/ui-web";
import { Pencil } from "lucide-react";
import { useCallback, useEffect, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import { AssetGrid, useAssetSelection } from "../../components/media/library/AssetGrid";
import { CreateProjectDialog } from "../../components/media/library/CreateProjectDialog";
import { collectionDateRange } from "../../components/media/library/LibraryChrome";
import { formatDuration } from "../../lib/media-library";
import { rpcErrorMessage } from "../../lib/media-studio";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
export function CollectionView({
navigate,
collectionId,
}: MediaViewProps & { collectionId: string }) {
const { t } = useLingui();
const [data, setData] = useState<{ collection: MediaCollection; assets: MediaAsset[] } | null>(
null,
);
const [error, setError] = useState<string | null>(null);
const [editing, setEditing] = useState<string | null>(null);
const [createOpen, setCreateOpen] = useState(false);
const selection = useAssetSelection();
useEffect(() => {
let active = true;
setData(null);
setError(null);
rpc.media.collections
.get({ collectionId })
.then((value) => {
if (active) setData(value);
})
.catch((cause) => {
if (active) setError(rpcErrorMessage(cause, t`Could not load this session.`));
});
return () => {
active = false;
};
}, [collectionId, t]);
const rename = useCallback(
async (title: string) => {
setEditing(null);
const trimmed = title.trim();
if (!data || !trimmed || trimmed === data.collection.title) return;
try {
const collection = await rpc.media.collections.rename({ collectionId, title: trimmed });
setData((current) => (current ? { ...current, collection } : current));
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not rename.`));
}
},
[collectionId, data, t],
);
if (!data) {
return (
<div className="grid flex-1 place-items-center px-8 text-center text-ui-body text-muted-foreground">
{error ?? <LoadingState label={t`Loading`} />}
</div>
);
}
const { collection, assets } = data;
const selectedIds = [...selection.selected];
const range = collectionDateRange(collection);
return (
<div data-testid="media-collection" className="flex min-h-0 flex-1 flex-col">
<div className="flex flex-wrap items-center gap-3 px-4 pt-3 pb-2">
<div className="min-w-0 flex-1">
{editing !== null ? (
<Input
autoFocus
aria-label={t`Name`}
value={editing}
maxLength={120}
className="max-w-md"
onChange={(event) => setEditing(event.target.value)}
onBlur={() => void rename(editing)}
onKeyDown={(event) => {
if (event.key === "Enter") void rename(editing);
if (event.key === "Escape") {
event.stopPropagation();
setEditing(null);
}
}}
/>
) : (
<div className="flex items-center gap-1">
<h2 className="truncate text-ui-title-sm font-semibold tracking-tight" dir="auto">
{collection.title}
</h2>
<Button
variant="ghost"
size="icon-xs"
aria-label={t`Rename`}
onClick={() => setEditing(collection.title)}
>
<Pencil />
</Button>
</div>
)}
<div className="text-ui-body-sm text-muted-foreground">
{range ? `${range} · ` : ""}
<Plural value={collection.assetCount} one="# item" other="# items" />
{collection.totalDurationMs > 0
? ` · ${formatDuration(collection.totalDurationMs)}`
: ""}
</div>
</div>
<div className="flex items-center gap-2">
{selectedIds.length > 0 ? (
<Button variant="ghost" size="sm" onClick={selection.clear}>
<Plural value={selectedIds.length} one="# selected" other="# selected" />
</Button>
) : null}
<Button
size="sm"
data-testid="media-create-project-open"
onClick={() => setCreateOpen(true)}
>
<Trans>Create project</Trans>
</Button>
</div>
</div>
{error ? (
<p role="alert" className="px-4 text-ui-body-sm text-destructive">
{error}
</p>
) : null}
<AssetGrid assets={assets} selection={selection} scrub />
<CreateProjectDialog
open={createOpen}
onOpenChange={setCreateOpen}
defaultTitle={collection.title}
collectionIds={selectedIds.length > 0 ? [] : [collection.id]}
assetIds={selectedIds}
onCreated={(projectId) => navigate({ name: "project", projectId })}
/>
</div>
);
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-editor slice replaces this view. */
export function EditorView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,294 @@
import { Plural, Trans, useLingui } from "@lingui/react/macro";
import type { MediaAsset, MediaCollection, MediaSource } from "@rakazo/contracts";
import { Button } from "@rakazo/ui-web";
import { Plus } from "lucide-react";
import { useCallback, useEffect, useRef, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import { AssetGrid, useAssetSelection } from "../../components/media/library/AssetGrid";
import { ConnectSourceDialog } from "../../components/media/library/ConnectSourceDialog";
import { CreateProjectDialog } from "../../components/media/library/CreateProjectDialog";
import {
CollectionCard,
FilterChips,
IndexingProgress,
LibraryModeToggle,
LibrarySearch,
} from "../../components/media/library/LibraryChrome";
import { usePhotosSync } from "../../components/media/library/usePhotosSync";
import { VirtualGrid } from "../../components/media/library/VirtualGrid";
import type { LibraryFilter } from "../../lib/media-library";
import { assetQueryFor, hasAssetQuery, INGEST_PENDING_STATES } from "../../lib/media-library";
import { usePolling } from "../../lib/media-poll";
import { rpcErrorMessage, useMediaStudio } from "../../lib/media-studio";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
const PAGE = 500;
type Paged<T> = { items: T[]; cursor: string | null; done: boolean; loading: boolean };
const emptyPage = <T,>(): Paged<T> => ({ items: [], cursor: null, done: false, loading: false });
/**
* Cursor-paged list that appends on demand. A new `key` starts over and drops
* pages still in flight for the previous key.
*/
function usePagedList<T>(
key: string,
load: (cursor: string | null) => Promise<{ items: T[]; nextCursor: string | null }>,
enabled: boolean,
) {
const [state, setState] = useState<Paged<T>>(emptyPage);
const [error, setError] = useState<unknown>(null);
const generation = useRef(0);
const busy = useRef(false);
const loadRef = useRef(load);
loadRef.current = load;
const stateRef = useRef(state);
stateRef.current = state;
const more = useCallback(async () => {
const current = stateRef.current;
if (busy.current || current.done) return;
busy.current = true;
const mine = generation.current;
setState((s) => ({ ...s, loading: true }));
try {
const page = await loadRef.current(current.cursor);
if (mine !== generation.current) return;
setError(null);
setState((s) => ({
items: [...s.items, ...page.items],
cursor: page.nextCursor,
done: page.nextCursor === null,
loading: false,
}));
} catch (cause) {
if (mine !== generation.current) return;
setError(cause);
setState((s) => ({ ...s, loading: false, done: true }));
} finally {
if (mine === generation.current) busy.current = false;
}
}, []);
const reset = useCallback(() => {
generation.current += 1;
busy.current = false;
stateRef.current = emptyPage();
setState(emptyPage());
setError(null);
}, []);
useEffect(() => {
void key;
reset();
if (enabled) void more();
}, [key, enabled, reset, more]);
const restart = useCallback(() => {
reset();
void more();
}, [reset, more]);
return { ...state, error, more, reset: restart };
}
export function LibraryView({
navigate,
mode = "sessions",
}: MediaViewProps & { mode?: "sessions" | "themes" }) {
const { t } = useLingui();
const { canLinkFolder } = useMediaStudio();
const [filters, setFilters] = useState<ReadonlySet<LibraryFilter>>(() => new Set());
const [text, setText] = useState("");
const [debouncedText, setDebouncedText] = useState("");
const [sources, setSources] = useState<MediaSource[] | null>(null);
const [pending, setPending] = useState(false);
const [connectOpen, setConnectOpen] = useState(false);
const [createOpen, setCreateOpen] = useState(false);
const selection = useAssetSelection();
useEffect(() => {
const timer = setTimeout(() => setDebouncedText(text), 250);
return () => clearTimeout(timer);
}, [text]);
const searching = hasAssetQuery(filters, debouncedText);
const query = assetQueryFor(filters, debouncedText);
const queryKey = JSON.stringify(query);
const kind = mode === "themes" ? "theme" : "session";
const collections = usePagedList<MediaCollection>(
kind,
(cursor) => rpc.media.collections.list({ kind, cursor, limit: PAGE }),
!searching,
);
const assets = usePagedList<MediaAsset>(
queryKey,
(cursor) => rpc.media.assets.search({ ...query, cursor, limit: PAGE }),
searching,
);
const refreshStatus = useCallback(async () => {
const [list, work] = await Promise.all([
rpc.media.sources.list({}),
rpc.media.assets.search({ ingestStates: [...INGEST_PENDING_STATES], cursor: null, limit: 1 }),
]);
setSources(list);
setPending(work.items.length > 0);
}, []);
const photos = usePhotosSync(
useCallback(() => {
void refreshStatus();
collections.reset();
}, [refreshStatus, collections.reset]),
);
useEffect(() => {
void refreshStatus().catch(() => setSources([]));
}, [refreshStatus]);
const indexing =
photos.syncing || pending || (sources ?? []).some((source) => source.status === "syncing");
usePolling(refreshStatus, indexing);
// Indexing just finished: the grouping changed underneath us.
const wasIndexing = useRef(indexing);
useEffect(() => {
if (wasIndexing.current && !indexing) {
if (searching) assets.reset();
else collections.reset();
}
wasIndexing.current = indexing;
}, [indexing, searching, assets.reset, collections.reset]);
const noSources = sources !== null && sources.length === 0 && !photos.syncing;
const selectedIds = [...selection.selected];
return (
<div data-testid="media-library" className="flex min-h-0 flex-1 flex-col">
<div className="flex flex-wrap items-center gap-2 px-4 pt-3 pb-2">
<LibraryModeToggle
mode={mode}
onChange={(next) =>
navigate(next === "people" ? { name: "people" } : { name: "library", mode: next })
}
/>
<LibrarySearch value={text} onChange={setText} />
<FilterChips value={filters} onChange={setFilters} />
<div className="ml-auto flex items-center gap-2">
{selectedIds.length > 0 ? (
<>
<Button variant="ghost" size="sm" onClick={selection.clear}>
<Plural value={selectedIds.length} one="# selected" other="# selected" />
</Button>
<Button size="sm" onClick={() => setCreateOpen(true)}>
<Trans>Create project</Trans>
</Button>
</>
) : null}
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Sources`}
onClick={() => setConnectOpen(true)}
>
<Plus />
</Button>
</div>
</div>
{indexing ? (
<IndexingProgress
done={photos.progress?.uploadsDone}
total={photos.progress?.uploadsTotal}
/>
) : null}
{sources === null ? (
<div className="grid flex-1 place-items-center">
<LoadingState label={t`Loading`} />
</div>
) : noSources ? (
<div className="grid flex-1 place-items-center px-8">
<div className="flex flex-col items-center gap-3 text-center">
<p className="text-ui-body text-muted-foreground">
<Trans>Connect your photos and videos.</Trans>
</p>
<Button data-testid="media-connect-source-open" onClick={() => setConnectOpen(true)}>
<Trans>Connect a source</Trans>
</Button>
</div>
</div>
) : searching ? (
assets.items.length === 0 && !assets.loading ? (
<EmptyMessage error={assets.error}>
<Trans>Nothing matches.</Trans>
</EmptyMessage>
) : (
<AssetGrid
assets={assets.items}
selection={selection}
scrub
onNearEnd={() => void assets.more()}
/>
)
) : collections.items.length === 0 && !collections.loading ? (
indexing ? null : (
<EmptyMessage error={collections.error}>
{mode === "themes" ? <Trans>No themes yet.</Trans> : <Trans>No sessions yet.</Trans>}
</EmptyMessage>
)
) : (
<VirtualGrid
testId="media-collection-grid"
label={mode === "themes" ? t`Themes` : t`Sessions`}
count={collections.items.length}
getKey={(index) => collections.items[index]?.id ?? index}
minTileWidth={220}
gap={16}
tileAspect={10 / 16}
extraHeight={44}
onNearEnd={() => void collections.more()}
renderItem={(index) => {
const collection = collections.items[index];
if (!collection) return null;
return (
<CollectionCard
collection={collection}
onOpen={() => navigate({ name: "collection", collectionId: collection.id })}
/>
);
}}
/>
)}
<ConnectSourceDialog
open={connectOpen}
onOpenChange={setConnectOpen}
sources={sources ?? []}
canLinkFolder={canLinkFolder}
photos={photos}
onChanged={() => void refreshStatus()}
/>
<CreateProjectDialog
open={createOpen}
onOpenChange={setCreateOpen}
defaultTitle={debouncedText.trim() || t`New project`}
collectionIds={[]}
assetIds={selectedIds}
onCreated={(projectId) => navigate({ name: "project", projectId })}
/>
</div>
);
}
function EmptyMessage({ error, children }: { error: unknown; children: React.ReactNode }) {
const { t } = useLingui();
return (
<div className="grid flex-1 place-items-center px-8 text-center text-ui-body text-muted-foreground">
{error ? rpcErrorMessage(error, t`Could not load the library.`) : children}
</div>
);
}

View File

@ -0,0 +1,192 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type { MediaProject } from "@rakazo/contracts";
import {
Button,
Dialog,
DialogClose,
DialogContent,
DialogTitle,
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuTrigger,
Tabs,
TabsList,
TabsTrigger,
} from "@rakazo/ui-web";
import { ChevronDown, ChevronLeft, X } from "lucide-react";
import { useCallback, useEffect, useMemo, useState } from "react";
import { MediaStudioContext, useMediaCapabilities } from "../../lib/media-studio";
import { rpc } from "../../lib/rpc";
import { BrandKitView } from "./BrandKitView";
import { CollectionView } from "./CollectionView";
import { EditorView } from "./EditorView";
import { LibraryView } from "./LibraryView";
import type { MediaSection, MediaView } from "./nav";
import { pushView, sectionOf } from "./nav";
import { PeopleView } from "./PeopleView";
import { ProjectView } from "./ProjectView";
import { ResultsView } from "./ResultsView";
import { ReviewView } from "./ReviewView";
function ViewSwitch({
view,
navigate,
onClose,
}: {
view: MediaView;
navigate: (view: MediaView) => void;
onClose: () => void;
}) {
const props = { navigate, onClose };
switch (view.name) {
case "library":
return <LibraryView {...props} mode={view.mode} />;
case "collection":
return <CollectionView {...props} collectionId={view.collectionId} />;
case "people":
return <PeopleView {...props} />;
case "results":
return <ResultsView {...props} />;
case "project":
return <ProjectView {...props} projectId={view.projectId} />;
case "editor":
return <EditorView {...props} projectId={view.projectId} planId={view.planId} />;
case "review":
return <ReviewView {...props} projectId={view.projectId} renderId={view.renderId} />;
case "brandKit":
return <BrandKitView {...props} brandKitId={view.brandKitId} />;
}
}
function ProjectsMenu({
navigate,
refreshKey,
}: {
navigate: (view: MediaView) => void;
refreshKey: unknown;
}) {
const [projects, setProjects] = useState<MediaProject[]>([]);
useEffect(() => {
void refreshKey;
let active = true;
rpc.media.projects
.list({ cursor: null, limit: 20 })
.then((page) => {
if (active) setProjects(page.items.filter((project) => project.status !== "archived"));
})
.catch(() => undefined);
return () => {
active = false;
};
}, [refreshKey]);
if (projects.length === 0) return null;
return (
<DropdownMenu>
<DropdownMenuTrigger
render={<Button variant="ghost" size="sm" data-testid="media-projects-menu" />}
>
<Trans>Projects</Trans>
<ChevronDown data-icon="inline-end" aria-hidden="true" />
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="max-h-80 w-64 overflow-y-auto">
{projects.map((project) => (
<DropdownMenuItem
key={project.id}
onClick={() => navigate({ name: "project", projectId: project.id })}
>
<span className="truncate" dir="auto">
{project.title}
</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
);
}
/** Media Studio surface: a full-window layer with its own back stack. */
export function MediaStudio({
onClose,
canLinkFolder = false,
}: {
onClose: () => void;
/** Deployment owner only: folder sources live on the host. */
canLinkFolder?: boolean;
}) {
const { t } = useLingui();
const capabilities = useMediaCapabilities();
const [stack, setStack] = useState<MediaView[]>([{ name: "library" }]);
const view = stack.at(-1) ?? { name: "library" };
const section = sectionOf(view);
const navigate = useCallback(
(next: MediaView) => setStack((current) => pushView(current, next)),
[],
);
const back = useCallback(
() => setStack((current) => (current.length > 1 ? current.slice(0, -1) : current)),
[],
);
const env = useMemo(() => ({ capabilities, canLinkFolder }), [capabilities, canLinkFolder]);
const notConfigured = capabilities !== null && !capabilities.configured;
return (
<Dialog
open
onOpenChange={(open) => {
if (!open) onClose();
}}
>
<DialogContent
data-testid="media-studio"
data-view={view.name}
showCloseButton={false}
className="inset-0 top-0 left-0 flex h-full w-full max-w-none translate-x-0 translate-y-0 flex-col gap-0 rounded-none bg-background p-0 text-foreground ring-0 sm:max-w-none"
>
<header className="flex items-center gap-2 border-b border-sidebar-border px-3 py-2.5 md:px-[18px]">
{stack.length > 1 ? (
<Button variant="ghost" size="icon-sm" aria-label={t`Back`} onClick={back}>
<ChevronLeft />
</Button>
) : null}
<DialogTitle className="text-ui-body font-medium text-foreground">
<Trans>Media</Trans>
</DialogTitle>
<div className="flex min-w-0 flex-1 justify-center">
<Tabs
value={section ?? ""}
onValueChange={(value) => navigate({ name: value as MediaSection })}
>
<TabsList variant="line" aria-label={t`Media Studio`}>
<TabsTrigger value="library">
<Trans>Library</Trans>
</TabsTrigger>
<TabsTrigger value="results">
<Trans>Results</Trans>
</TabsTrigger>
<TabsTrigger value="brandKit">
<Trans>Brand kit</Trans>
</TabsTrigger>
</TabsList>
</Tabs>
</div>
<ProjectsMenu navigate={navigate} refreshKey={view} />
<DialogClose aria-label={t`Close`} render={<Button variant="ghost" size="icon-sm" />}>
<X />
</DialogClose>
</header>
<MediaStudioContext.Provider value={env}>
<main className="flex min-h-0 flex-1 flex-col">
{notConfigured ? (
<div className="grid flex-1 place-items-center px-8 text-center text-ui-body text-muted-foreground">
<Trans>Media Studio is not set up on this server.</Trans>
</div>
) : (
<ViewSwitch view={view} navigate={navigate} onClose={onClose} />
)}
</main>
</MediaStudioContext.Provider>
</DialogContent>
</Dialog>
);
}

View File

@ -0,0 +1,167 @@
import { Plural, Trans, useLingui } from "@lingui/react/macro";
import type { MediaPerson } from "@rakazo/contracts";
import { Button } from "@rakazo/ui-web";
import { useCallback, useEffect, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import { ConfirmDialog } from "../../components/media/library/ConfirmDialog";
import { ConsentDialog } from "../../components/media/library/ConsentDialog";
import { LibraryModeToggle } from "../../components/media/library/LibraryChrome";
import { PersonCard } from "../../components/media/library/PersonCard";
import { rpcErrorMessage, useMediaStudio } from "../../lib/media-studio";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
export function PeopleView({ navigate }: MediaViewProps) {
const { t } = useLingui();
const { capabilities } = useMediaStudio();
const [people, setPeople] = useState<MediaPerson[] | null>(null);
const [error, setError] = useState<string | null>(null);
const [selected, setSelected] = useState<ReadonlySet<string>>(() => new Set());
const [consentFor, setConsentFor] = useState<MediaPerson | null>(null);
const [removing, setRemoving] = useState<MediaPerson | null>(null);
const [merging, setMerging] = useState(false);
const syntheticAvailable = capabilities?.operations.synthesizeIdentity?.available === true;
const load = useCallback(async () => {
try {
setPeople(await rpc.media.people.list({}));
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not load people.`));
setPeople((current) => current ?? []);
}
}, [t]);
useEffect(() => {
void load();
}, [load]);
const replace = (person: MediaPerson) =>
setPeople((current) => current?.map((row) => (row.id === person.id ? person : row)) ?? null);
const act = async (action: () => Promise<unknown>, fallback: string) => {
setError(null);
try {
await action();
} catch (cause) {
setError(rpcErrorMessage(cause, fallback));
}
};
const merge = () =>
act(async () => {
const merged = await rpc.media.people.merge({ personIds: [...selected] });
setSelected(new Set());
await load();
replace(merged);
}, t`Could not merge.`);
return (
<div data-testid="media-people" className="flex min-h-0 flex-1 flex-col">
<div className="flex flex-wrap items-center gap-2 px-4 pt-3 pb-2">
<LibraryModeToggle
mode="people"
onChange={(next) => {
if (next !== "people") navigate({ name: "library", mode: next });
}}
/>
{selected.size >= 2 ? (
<div className="ml-auto flex items-center gap-2">
<Button variant="ghost" size="sm" onClick={() => setSelected(new Set())}>
<Plural value={selected.size} one="# selected" other="# selected" />
</Button>
<Button size="sm" onClick={() => setMerging(true)}>
<Trans>Merge</Trans>
</Button>
</div>
) : null}
</div>
{error ? (
<p role="alert" className="px-4 pb-2 text-ui-body-sm text-destructive">
{error}
</p>
) : null}
{people === null ? (
<div className="grid flex-1 place-items-center">
<LoadingState label={t`Loading`} />
</div>
) : people.length === 0 ? (
<div className="grid flex-1 place-items-center px-8 text-center text-ui-body text-muted-foreground">
<Trans>No people found yet.</Trans>
</div>
) : (
<div className="rk-scroll min-h-0 flex-1 overflow-y-auto px-4 pb-4">
<div className="grid grid-cols-[repeat(auto-fill,minmax(260px,1fr))] gap-3">
{people.map((person) => (
<PersonCard
key={person.id}
person={person}
syntheticAvailable={syntheticAvailable}
selected={selected.has(person.id)}
onSelect={(value) =>
setSelected((current) => {
const next = new Set(current);
if (value) next.add(person.id);
else next.delete(person.id);
return next;
})
}
onUpdate={(patch) =>
void act(
async () =>
replace(await rpc.media.people.update({ personId: person.id, ...patch })),
t`Could not save.`,
)
}
onConsent={() => setConsentFor(person)}
onRemove={() => setRemoving(person)}
/>
))}
</div>
</div>
)}
<ConsentDialog
person={consentFor}
onOpenChange={(open) => {
if (!open) setConsentFor(null);
}}
onSaved={replace}
/>
<ConfirmDialog
open={merging}
title={<Plural value={selected.size} one="Merge # person?" other="Merge # people?" />}
description={<Trans>Their appearances become one person.</Trans>}
action={<Trans>Merge</Trans>}
onCancel={() => setMerging(false)}
onConfirm={() => {
setMerging(false);
void merge();
}}
/>
<ConfirmDialog
open={removing !== null}
destructive
title={<Trans>Delete face data of {removing?.displayName ?? ""}?</Trans>}
description={
<Trans>Face tracks, recognition data and marks of this person are deleted.</Trans>
}
action={<Trans>Delete</Trans>}
onCancel={() => setRemoving(null)}
onConfirm={() => {
const target = removing;
setRemoving(null);
if (!target) return;
void act(async () => {
await rpc.media.people.remove({ personId: target.id });
setPeople((current) => current?.filter((row) => row.id !== target.id) ?? null);
setSelected((current) => {
const next = new Set(current);
next.delete(target.id);
return next;
});
}, t`Could not delete.`);
}}
/>
</div>
);
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-editor slice replaces this view. */
export function ProjectView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,307 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
FunnelRow,
MediaProject,
MediaPublication,
SegmentFunnelRow,
} from "@rakazo/contracts";
import {
Button,
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
NativeSelect,
NativeSelectOption,
} from "@rakazo/ui-web";
import { MoreHorizontal, Upload } from "lucide-react";
import { useCallback, useEffect, useMemo, useState } from "react";
import { LoadingState, SuccessPop } from "../../components/ai/primitives";
import { AssetThumb } from "../../components/media/library/AssetTile";
import { ConfirmDialog } from "../../components/media/library/ConfirmDialog";
import { CsvImportDialog } from "../../components/media/library/CsvImportDialog";
import { Funnel } from "../../components/media/library/Funnel";
import { MetricRecordDialog } from "../../components/media/library/MetricRecordDialog";
import { formatDuration, formatMoney, funnelTotals } from "../../lib/media-library";
import { rpcErrorMessage } from "../../lib/media-studio";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
const ALL = "";
type Data = {
funnel: FunnelRow[];
segments: SegmentFunnelRow[];
publications: MediaPublication[];
};
export function ResultsView({ navigate }: MediaViewProps) {
const { t, i18n } = useLingui();
const [projects, setProjects] = useState<MediaProject[]>([]);
const [projectId, setProjectId] = useState(ALL);
const [data, setData] = useState<Data | null>(null);
const [error, setError] = useState<string | null>(null);
const [importOpen, setImportOpen] = useState(false);
const [recordOpen, setRecordOpen] = useState(false);
const [resetOpen, setResetOpen] = useState(false);
const [resetDone, setResetDone] = useState(false);
useEffect(() => {
rpc.media.projects
.list({ cursor: null, limit: 100 })
.then((page) => setProjects(page.items))
.catch(() => undefined);
}, []);
const load = useCallback(async () => {
const scope = projectId === ALL ? {} : { projectId };
try {
const [funnel, segments, publications] = await Promise.all([
rpc.media.metrics.funnel(scope),
rpc.media.metrics.segments(scope),
rpc.media.publications.list({ ...scope, cursor: null, limit: 500 }),
]);
setData({ funnel, segments, publications: publications.items });
setError(null);
} catch (cause) {
setError(rpcErrorMessage(cause, t`Could not load results.`));
setData((current) => current ?? { funnel: [], segments: [], publications: [] });
}
}, [projectId, t]);
useEffect(() => {
void load();
}, [load]);
const totals = useMemo(() => funnelTotals(data?.funnel ?? []), [data]);
const projectTitle = useMemo(
() => new Map(projects.map((project) => [project.id, project.title])),
[projects],
);
const locale = i18n.locale;
const number = (value: number) => new Intl.NumberFormat(locale).format(value);
const hasPublications = (data?.publications.length ?? 0) > 0;
return (
<div data-testid="media-results" className="flex min-h-0 flex-1 flex-col">
<div className="flex flex-wrap items-center gap-2 px-4 pt-3 pb-2">
{projects.length > 0 ? (
<NativeSelect
size="sm"
aria-label={t`Project`}
value={projectId}
onChange={(event) => setProjectId(event.target.value)}
>
<NativeSelectOption value={ALL}>{t`All projects`}</NativeSelectOption>
{projects.map((project) => (
<NativeSelectOption key={project.id} value={project.id}>
{project.title}
</NativeSelectOption>
))}
</NativeSelect>
) : null}
<div className="ml-auto flex items-center gap-2">
{resetDone ? <SuccessPop label={t`Reset`} /> : null}
<Button size="sm" variant="outline" onClick={() => setImportOpen(true)}>
<Upload data-icon="inline-start" aria-hidden="true" />
<Trans>Import CSV</Trans>
</Button>
<DropdownMenu>
<DropdownMenuTrigger
render={<Button variant="ghost" size="icon-sm" aria-label={t`More`} />}
>
<MoreHorizontal />
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem disabled={!hasPublications} onClick={() => setRecordOpen(true)}>
<Trans>Enter results</Trans>
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem variant="destructive" onClick={() => setResetOpen(true)}>
<Trans>Reset teaser model</Trans>
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
</div>
{error ? (
<p role="alert" className="px-4 pb-2 text-ui-body-sm text-destructive">
{error}
</p>
) : null}
{data === null ? (
<div className="grid flex-1 place-items-center">
<LoadingState label={t`Loading`} />
</div>
) : data.funnel.length === 0 ? (
<div className="grid flex-1 place-items-center px-8 text-center text-ui-body text-muted-foreground">
<Trans>No results yet.</Trans>
</div>
) : (
<div className="rk-scroll flex min-h-0 flex-1 flex-col gap-6 overflow-y-auto px-4 pb-6">
<Funnel totals={totals} />
<section aria-label={t`Publications`}>
<h3 className="mb-2 text-ui-body-sm font-medium text-muted-foreground">
<Trans>Publications</Trans>
</h3>
<div className="overflow-x-auto">
<table data-testid="media-results-publications" className="w-full text-ui-body-sm">
<thead className="text-left text-muted-foreground">
<tr className="border-b border-border">
<th className="py-1.5 pr-3 font-medium">
<Trans>Project</Trans>
</th>
<th className="py-1.5 pr-3 font-medium">
<Trans>Platform</Trans>
</th>
<th className="py-1.5 pr-3 font-medium">
<Trans>Variant</Trans>
</th>
<NumberHead>
<Trans>Views</Trans>
</NumberHead>
<NumberHead>
<Trans>Clicks</Trans>
</NumberHead>
<NumberHead>
<Trans>Purchases</Trans>
</NumberHead>
<NumberHead>
<Trans>Revenue</Trans>
</NumberHead>
</tr>
</thead>
<tbody>
{data.funnel.map((row) => (
<tr
key={`${row.publicationId}-${row.currency}`}
className="border-b border-border/60"
>
<td className="py-1.5 pr-3">
<button
type="button"
className="max-w-48 truncate text-left hover:underline"
onClick={() => navigate({ name: "project", projectId: row.projectId })}
dir="auto"
>
{projectTitle.get(row.projectId) ?? "–"}
</button>
</td>
<td className="py-1.5 pr-3">{row.profileId}</td>
<td className="py-1.5 pr-3 text-muted-foreground">
{[row.variant, row.thumbnailVariant].filter(Boolean).join(" · ") || "–"}
</td>
<NumberCell>{number(row.views)}</NumberCell>
<NumberCell>{number(row.clicks)}</NumberCell>
<NumberCell>{number(row.conversions)}</NumberCell>
<NumberCell>{formatMoney(row.revenueCents, row.currency, locale)}</NumberCell>
</tr>
))}
</tbody>
</table>
</div>
</section>
{data.segments.length > 0 ? (
<section aria-label={t`Teaser segments`}>
<h3 className="mb-2 text-ui-body-sm font-medium text-muted-foreground">
<Trans>Teaser segments</Trans>
</h3>
<div className="overflow-x-auto">
<table data-testid="media-results-segments" className="w-full text-ui-body-sm">
<thead className="text-left text-muted-foreground">
<tr className="border-b border-border">
<th className="py-1.5 pr-3 font-medium">
<Trans>Segment</Trans>
</th>
<NumberHead>
<Trans>Uses</Trans>
</NumberHead>
<NumberHead>
<Trans>Views</Trans>
</NumberHead>
<NumberHead>
<Trans>Clicks</Trans>
</NumberHead>
<NumberHead>
<Trans>Purchases</Trans>
</NumberHead>
<NumberHead>
<Trans>Revenue</Trans>
</NumberHead>
</tr>
</thead>
<tbody>
{data.segments.map((row) => (
<tr
key={`${row.segmentId}-${row.currency}`}
className="border-b border-border/60"
>
<td className="py-1.5 pr-3">
<div className="flex items-center gap-2">
<div className="h-7 w-12 shrink-0 overflow-hidden rounded bg-muted">
<AssetThumb assetId={row.assetId} hasThumbnail />
</div>
<span className="font-mono tabular-nums text-muted-foreground">
{formatDuration(row.startMs)}–{formatDuration(row.endMs)}
</span>
</div>
</td>
<NumberCell>{number(row.publications)}</NumberCell>
<NumberCell>{number(row.views)}</NumberCell>
<NumberCell>{number(row.clicks)}</NumberCell>
<NumberCell>{number(row.conversions)}</NumberCell>
<NumberCell>
{formatMoney(row.revenueCents, row.currency, locale)}
</NumberCell>
</tr>
))}
</tbody>
</table>
</div>
</section>
) : null}
</div>
)}
<CsvImportDialog
open={importOpen}
onOpenChange={setImportOpen}
onImported={() => void load()}
/>
<MetricRecordDialog
open={recordOpen}
onOpenChange={setRecordOpen}
publications={data?.publications ?? []}
onRecorded={() => void load()}
/>
<ConfirmDialog
open={resetOpen}
destructive
title={<Trans>Reset teaser model?</Trans>}
description={<Trans>Teaser picks go back to the default weights.</Trans>}
action={<Trans>Reset</Trans>}
onCancel={() => setResetOpen(false)}
onConfirm={() => {
setResetOpen(false);
setResetDone(false);
rpc.media.metrics
.resetTeaserModel({})
.then(() => setResetDone(true))
.catch((cause) => setError(rpcErrorMessage(cause, t`Could not reset.`)));
}}
/>
</div>
);
}
function NumberHead({ children }: { children: React.ReactNode }) {
return <th className="py-1.5 pl-3 text-right font-medium">{children}</th>;
}
function NumberCell({ children }: { children: React.ReactNode }) {
return <td className="py-1.5 pl-3 text-right font-mono tabular-nums">{children}</td>;
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-editor slice replaces this view. */
export function ReviewView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,47 @@
/** Screens inside Media Studio. Every view receives `navigate` to move between them. */
export type MediaView =
| { name: "library"; mode?: "sessions" | "themes" }
| { name: "collection"; collectionId: string }
| { name: "people" }
| { name: "results" }
| { name: "project"; projectId: string }
| { name: "editor"; projectId: string; planId: string }
| { name: "review"; projectId: string; renderId: string }
| { name: "brandKit"; brandKitId?: string };
export interface MediaViewProps {
navigate: (view: MediaView) => void;
}
/** Top-level sections: opening one starts a fresh back stack. */
export type MediaSection = "library" | "results" | "brandKit";
export function sectionOf(view: MediaView): MediaSection | null {
switch (view.name) {
case "library":
case "people":
case "collection":
return "library";
case "results":
return "results";
case "brandKit":
return "brandKit";
default:
return null;
}
}
export function isRootView(view: MediaView): boolean {
return (
view.name === "library" ||
view.name === "people" ||
view.name === "results" ||
(view.name === "brandKit" && !view.brandKitId)
);
}
/** Push `next` onto the back stack; root views replace it. */
export function pushView(stack: readonly MediaView[], next: MediaView): MediaView[] {
if (isRootView(next)) return [next];
return [...stack, next];
}

View File

@ -10,6 +10,8 @@ import {
createCloudAgentConnection,
createConnectorStack,
createJobReconciler,
createMediaJobHandlers,
createMediaServices,
createMessagingContextLoader,
createPostgresReconciliationLeadership,
createProfileConsolidationScheduler,
@ -31,6 +33,7 @@ import {
LocalArtifactStore,
McpConnector,
McpOAuthBroker,
mediaConfigFromEnv,
messagingEnvFromProcess,
messagingPlatformsFromEnv,
PiAgentRuntime,
@ -201,6 +204,25 @@ async function main() {
cloudAgent,
});
const mediaAbort = new AbortController();
const mediaServices = createMediaServices({
config: mediaConfigFromEnv(process.env),
prisma,
jobs,
artifacts,
runtime,
realtime,
resolveModel: (scope) => executor.resolveModel(scope),
});
const mediaJobs = mediaServices
? createMediaJobHandlers(mediaServices, {
signal: mediaAbort.signal,
limits: {
gpu: parsePositiveInteger(process.env.MEDIA_GPU_CONCURRENCY, 1),
cpu: parsePositiveInteger(process.env.MEDIA_CPU_CONCURRENCY, 2),
},
})
: null;
const jobHandlers = createBackgroundJobHandlers({
executor,
prisma,
@ -208,6 +230,7 @@ async function main() {
home,
jobs,
events,
...(mediaJobs ? { media: mediaJobs.handlers } : {}),
workerId: process.pid.toString(),
runtime,
secretStore: secrets,
@ -235,6 +258,9 @@ async function main() {
await new Promise((resolve) => setTimeout(resolve, databaseCapacityBackoffMs(attempt)));
}
}
if (mediaJobs) {
void mediaJobs.resume().catch((error) => logger.error("media ingest resume failed", error));
}
const reconciler = createJobReconciler({
prisma,
jobs,
@ -253,6 +279,7 @@ async function main() {
stopping = true;
try {
await reconciler.stop();
mediaAbort.abort();
await jobHost.stop();
await jobs.close();
await realtime.close();

View File

@ -129,6 +129,28 @@ services:
data-init:
condition: service_completed_successfully
media-worker:
# Optional Media Studio sidecar: `docker compose --profile media up`.
# No published port; the API and worker reach it as http://media-worker:8790.
profiles: ["media"]
build:
context: ../media-worker
args:
ACCELERATOR: ${MEDIA_ACCELERATOR:-cpu}
image: rakazo/media-worker:local
environment:
MEDIA_WORKER_HOST: "0.0.0.0"
MEDIA_WORKER_PORT: "8790"
MEDIA_WORKER_TOKEN: ${MEDIA_WORKER_TOKEN:?Set MEDIA_WORKER_TOKEN in .env}
MEDIA_ROOT: /data/media
MEDIA_MODELS_DIR: /models
HTTP_PROXY: ${HTTP_PROXY:-${http_proxy:-}}
HTTPS_PROXY: ${HTTPS_PROXY:-${https_proxy:-}}
NO_PROXY: ${NO_PROXY:-${no_proxy:-}}
volumes:
- ../../data/media:/data/media
- media-models:/models
web:
build:
context: ../..
@ -144,4 +166,5 @@ services:
- api
volumes:
media-models:
pgdata:

View File

@ -0,0 +1,7 @@
.pytest_cache
.ruff_cache
.mypy_cache
.venv
.venv-ml
**/__pycache__
**/*.pyc

5
infra/media-worker/.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
.pytest_cache/
.venv/
.venv
.venv-ml
__pycache__/

View File

@ -0,0 +1,68 @@
# syntax=docker/dockerfile:1.7
# Media Studio inference sidecar.
# docker build --build-arg ACCELERATOR=cpu|cuda|rocm -t rakazo/media-worker infra/media-worker
# docker build --target test -t rakazo/media-worker:test infra/media-worker # offline pytest
# Model weights are not baked in; they download into /models on first use.
# Behind a TLS-intercepting proxy, pass its CA with --secret id=pip_ca,src=<ca.pem>.
ARG ACCELERATOR=cpu
FROM python:3.11-slim-bookworm@sha256:2333bd330d12de02514770b3585cad313644316047cdee24a7acfdece6de6efb AS runtime
ARG ACCELERATOR
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
TQDM_DISABLE=1 \
MEDIA_ACCELERATOR=${ACCELERATOR} \
MEDIA_MODELS_DIR=/models \
MEDIA_WORKER_PORT=8790
RUN case "${ACCELERATOR}" in cpu|cuda|rocm) ;; *) echo "ACCELERATOR must be cpu, cuda or rocm" >&2; exit 1 ;; esac \
&& apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg libsndfile1 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /srv/media-worker
COPY requirements.txt requirements-ml-cpu.txt requirements-ml-cuda.txt requirements-ml-rocm.txt ./
# faster-whisper depends on the CPU onnxruntime wheel; the CUDA image swaps in
# onnxruntime-gpu. SAM2 is an sdist built against the installed torch, without
# its optional CUDA extension.
RUN --mount=type=secret,id=pip_ca,required=false \
if [ -s /run/secrets/pip_ca ]; then export PIP_CERT=/run/secrets/pip_ca; fi \
&& pip install -r requirements.txt -r "requirements-ml-${ACCELERATOR}.txt" \
&& if [ "${ACCELERATOR}" = "cuda" ]; then \
pip uninstall -y onnxruntime && pip install --no-deps --force-reinstall onnxruntime-gpu==1.30.0; \
fi \
&& SAM2_BUILD_CUDA=0 pip install --no-build-isolation --no-deps sam2==1.1.0 \
&& python -c "import cv2, scenedetect, librosa, onnxruntime, faster_whisper, torch, sam2, tokenizers"
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin media \
&& mkdir -p /models \
&& chown media:media /models
VOLUME ["/models"]
COPY app ./app
USER media
EXPOSE 8790
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD ["python", "-c", "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:' + os.environ.get('MEDIA_WORKER_PORT', '8790') + '/healthz', timeout=4)"]
CMD ["python", "-m", "app"]
# Offline test image: same runtime plus pytest; fake backends, no weights.
FROM runtime AS test
USER root
COPY requirements-dev.txt pytest.ini ./
RUN --mount=type=secret,id=pip_ca,required=false \
if [ -s /run/secrets/pip_ca ]; then export PIP_CERT=/run/secrets/pip_ca; fi \
&& pip install -r requirements-dev.txt
COPY tests ./tests
USER media
CMD ["python", "-m", "pytest", "-q"]
FROM runtime

View File

@ -0,0 +1,53 @@
# Licenses: models and libraries of the media sidecar
Checked October 2026 against the upstream LICENSE files and model cards linked
below. "Commercial use" means the published code or weights license permits it.
Training-data terms were not reviewed legally; where provenance is unclear the
backend is optional, never a default. Weights are downloaded at runtime from the
pinned revision (see `app/models.py`) and are never part of this repository or
the image.
## Models
| Name | Purpose | License | Commercial use | Default / optional | Source |
|---|---|---|---|---|---|
| OpenCV YuNet 2023mar | Face detection, 5-point landmarks (`detectFaces`, `trackFaces`, `faceLandmarks`) | MIT | yes | default | https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet |
| Whisper small (CTranslate2 conversion by Systran) | Transcription with word timestamps (`transcribe`) | MIT | yes | default | https://huggingface.co/Systran/faster-whisper-small, https://github.com/openai/whisper |
| SigLIP base patch16-224 (ONNX export by Xenova) | Frame embeddings (`embedFrames`); zero-shot content scores (`classifyContent`, experimental) | Apache-2.0 | yes | default | https://huggingface.co/google/siglip-base-patch16-224, https://huggingface.co/Xenova/siglip-base-patch16-224 |
| SAM 2.1 Hiera tiny | Click/box segmentation with video propagation (`segmentPrompted`, experimental) | Apache-2.0 | yes | default | https://github.com/facebookresearch/sam2, https://huggingface.co/facebook/sam2.1-hiera-tiny |
| Big-LaMa (ONNX export by Carve) | Per-frame inpainting with optical-flow smoothing (`inpaintVideo`, experimental) | Apache-2.0 | yes | default | https://github.com/advimman/lama, https://huggingface.co/Carve/LaMa-ONNX |
| OpenCV SFace 2021dec | Identity embeddings for local person clustering (`trackFaces.embedding`) | Apache-2.0 (weights); training-data provenance unverified | unclear | optional (`MEDIA_OPTIONAL_BACKENDS=faceIdentity`) | https://github.com/opencv/opencv_zoo/tree/main/models/face_recognition_sface |
| MediaPipe Face Landmarker | Dense face landmarks | Apache-2.0 | yes | not used: the `mediapipe` wheel pulls the GUI OpenCV build; YuNet's 5 points cover the partial modes | https://github.com/google-ai-edge/mediapipe |
| OpenCLIP (LAION weights) | Frame embeddings | MIT | yes | not used: SigLIP runs on ONNX Runtime without torch | https://github.com/mlfoundations/open_clip |
| NudeNet v3 | Nudity detection | AGPL-3.0 | copyleft, not shipped | not used | https://github.com/notAI-tech/NudeNet |
| ProPainter | Video inpainting | S-Lab License 1.0 (non-commercial) | no | not used; never default | https://github.com/sczhou/ProPainter |
| InsightFace model zoo (SCRFD, ArcFace, inswapper) | Face detection, recognition, swap | code MIT; models non-commercial research only | no | not used; never default | https://github.com/deepinsight/insightface |
| Synthetic identity generator + swap | `synthesizeIdentity` | none found with a commercial license | — | unavailable: always 501 `unsupported` | — |
## Methods without weights
| Name | Purpose | License | Commercial use | Default / optional | Source |
|---|---|---|---|---|---|
| PySceneDetect (AdaptiveDetector) | Shot detection (`detectScenes`) | BSD-3-Clause | yes | default | https://github.com/Breakthrough/PySceneDetect |
| IoU tracker (ByteTrack-style, in `app/ops/faces.py`) | Face tracking | this repository | yes | default | — |
| Spectral residual saliency (Hou & Zhang 2007, in `app/ops/saliency.py`) | `saliency` | this repository | yes | default | https://doi.org/10.1109/CVPR.2007.383267 |
| FFmpeg `ebur128`, `silencedetect` | Loudness, true peak, silences (`audioFeatures`) | LGPL-2.1+ / GPL-2+ (Debian build with libx264) | yes, as a separate process | default | https://ffmpeg.org/legal.html |
| librosa | Beat and tempo tracking (`audioFeatures`) | ISC | yes | default | https://github.com/librosa/librosa |
## Libraries
| Name | License | Commercial use |
|---|---|---|
| FastAPI | MIT | yes |
| Uvicorn | BSD-3-Clause | yes |
| Pydantic | MIT | yes |
| NumPy | BSD-3-Clause | yes |
| OpenCV (opencv-python-headless) | Apache-2.0 (wheel bundles FFmpeg under LGPL) | yes |
| soundfile | BSD-3-Clause | yes |
| PyTorch, torchvision | BSD-3-Clause (bundles permissive third-party code: Apache-2.0, MIT, BSL-1.0) | yes |
| ONNX Runtime | MIT | yes |
| faster-whisper | MIT | yes |
| CTranslate2 | MIT | yes |
| Hugging Face tokenizers | Apache-2.0 | yes |
| SAM 2 package (`sam2`) | Apache-2.0 | yes |
| Hydra, iopath | MIT | yes |

View File

@ -0,0 +1,74 @@
# media-worker
Local inference sidecar for Media Studio (Python 3.11, FastAPI). It implements
every `MediaInferenceProvider` operation (`packages/adapter-kit/src/media.ts`)
behind HTTP API v1; the TypeScript client is `packages/adapters/src/media-sidecar.ts`.
## API
- `GET /healthz` → `{"ok": true, "version": "..."}` (no token)
- `GET /v1/capabilities` → `MediaCapabilities`
- `POST /v1/<operation>` with the TypeScript request object (minus `signal`)
- Errors: `{"error": {"code", "message", "operation"}}` with
`bad_request`/`invalid_path` 400, `unauthorized` 401, `too_large` 413,
`unsupported` 501, `busy` 503, `internal` 500.
Every route but `/healthz` needs `Authorization: Bearer $MEDIA_WORKER_TOKEN`.
Paths must be absolute and resolve (symlinks included) inside `MEDIA_ROOT`.
A request stops when its client disconnects (between frames or chunks) or
after `MEDIA_MAX_SECONDS`.
## Configuration
| Variable | Default | |
|---|---|---|
| `MEDIA_WORKER_TOKEN` | — | required, ≥ 32 chars (`MEDIA_WORKER_INSECURE_DEV=1` lifts this for tests) |
| `MEDIA_ROOT` | — | required; mount it at the same absolute path as the API's `DATA_DIR/media` |
| `MEDIA_MODELS_DIR` | `/models` | weight cache (volume) |
| `MEDIA_WORKER_HOST` / `MEDIA_WORKER_PORT` | `127.0.0.1` / `8790` | |
| `MEDIA_GPU_CONCURRENCY` | `1` | model operations running at once |
| `MEDIA_CPU_CONCURRENCY` | `2` | scenes, redact, audio, saliency running at once |
| `MEDIA_QUEUE_SIZE` | `8` | waiting requests per gate before `503 busy` |
| `MEDIA_MAX_INPUT_BYTES` | 20 GiB | larger inputs → 413 |
| `MEDIA_MAX_SECONDS` | 14400 | longer inputs → 413; longer processing is cancelled |
| `MEDIA_MODEL_DOWNLOAD` | `1` | `0` = offline; missing weights → 501 |
| `MEDIA_OPTIONAL_BACKENDS` | empty | opt-in for backends that are not clean for commercial use, e.g. `faceIdentity` |
| `MEDIA_ACCELERATOR` | build arg | `cpu`, `cuda` or `rocm`; falls back to cpu when no GPU is visible |
## Models
Backends, licenses, URLs, SHA256 and sizes are pinned in `app/models.py`; the
license review is in `LICENSES.md`. Weights download on first use into
`MEDIA_MODELS_DIR/<backend>/`, are checked against the pinned SHA256 and size,
and are deleted on mismatch (the operation then reports 501). Warm the volume
ahead of time with `python -m app prefetch`.
`synthesizeIdentity` always answers 501: there is no commercially licensed
generator and swap pipeline, and the request schema rejects any extra field,
so a target-face image cannot be passed.
## Redaction geometry
Boxes interpolate linearly between keyframes; a single keyframe is a static
region for the whole clip. `paddingPct` grows the box by that share of its
size, `featherPct` fades the effect outward over that share of the shorter
side (the region itself stays fully covered). Partial modes use the first two
landmarks as eyes and the third as the nose. Output is FFV1 for `.mkv`,
otherwise x264 CRF 12; audio is copied and metadata dropped. Any frame error is
a 500 and nothing is left at `outPath`.
## Build and test
```sh
docker build --build-arg ACCELERATOR=cpu -t rakazo/media-worker infra/media-worker
docker build --target test -t rakazo/media-worker:test infra/media-worker
docker run --rm rakazo/media-worker:test # offline pytest, fake model backends
```
Without Docker: `pip install -r requirements.txt -r requirements-dev.txt` and
`python -m pytest` in this directory. Operations whose ML packages are missing
report `available: false`.
CUDA uses the torch cu126 wheels and `onnxruntime-gpu`. ROCm uses the torch
rocm7.1 wheels for SAM 2; ONNX Runtime and CTranslate2 have no ROCm wheels on
PyPI, so SigLIP, LaMa and Whisper run on the CPU there.

View File

@ -0,0 +1,3 @@
"""Local Media Studio inference sidecar."""
VERSION = "0.1.0"

View File

@ -0,0 +1,35 @@
"""Entry point: `python -m app` serves the API, `python -m app prefetch` warms the model volume."""
from __future__ import annotations
import logging
import sys
import uvicorn
from .config import ConfigError, Settings
from .main import create_app
from .models import Registry
def main(argv: list[str]) -> int:
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
try:
settings = Settings.from_env()
except ConfigError as error:
print(f"media-worker: {error}", file=sys.stderr)
return 2
if argv[1:] == ["prefetch"]:
Registry(settings).prefetch()
return 0
if argv[1:]:
print("usage: python -m app [prefetch]", file=sys.stderr)
return 2
uvicorn.run(
create_app(settings), host=settings.host, port=settings.port, log_level="info", access_log=False
)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

View File

@ -0,0 +1,22 @@
"""Bearer-token check for every route except /healthz."""
from __future__ import annotations
import hmac
from fastapi import Request
from .errors import MediaError
def require_token(request: Request) -> None:
expected: str = request.app.state.settings.token
header = request.headers.get("authorization", "")
scheme, _, presented = header.partition(" ")
if (
scheme.lower() != "bearer"
or not presented
or not expected
or not hmac.compare_digest(presented.strip().encode(), expected.encode())
):
raise MediaError("unauthorized", "missing or invalid bearer token")

View File

@ -0,0 +1,80 @@
"""Sidecar settings, read once from the environment."""
from __future__ import annotations
import os
from dataclasses import dataclass, field
MIN_TOKEN_LENGTH = 32
GIB = 1024**3
class ConfigError(Exception):
pass
def _int(env: dict[str, str], name: str, default: int, minimum: int = 0) -> int:
raw = env.get(name, "").strip()
if not raw:
return default
try:
value = int(raw)
except ValueError as error:
raise ConfigError(f"{name} must be an integer") from error
if value < minimum:
raise ConfigError(f"{name} must be >= {minimum}")
return value
@dataclass(frozen=True)
class Settings:
token: str
media_root: str
models_dir: str
host: str = "127.0.0.1"
port: int = 8790
accelerator: str = "cpu"
gpu_concurrency: int = 1
cpu_concurrency: int = 2
queue_size: int = 8
max_input_bytes: int = 20 * GIB
max_seconds: int = 4 * 3600
allow_model_download: bool = True
# Backends that are not licensed for commercial use stay off unless named here.
optional_backends: frozenset[str] = field(default_factory=frozenset)
insecure_dev: bool = False
@staticmethod
def from_env(env: dict[str, str] | None = None) -> Settings:
env = dict(os.environ if env is None else env)
insecure = env.get("MEDIA_WORKER_INSECURE_DEV", "") == "1"
token = env.get("MEDIA_WORKER_TOKEN", "")
if len(token) < MIN_TOKEN_LENGTH and not insecure:
raise ConfigError(f"MEDIA_WORKER_TOKEN must be at least {MIN_TOKEN_LENGTH} characters")
media_root = env.get("MEDIA_ROOT", "").strip()
if not media_root or not os.path.isabs(media_root):
raise ConfigError("MEDIA_ROOT must be an absolute path")
if not os.path.isdir(media_root):
raise ConfigError("MEDIA_ROOT does not exist")
accelerator = env.get("MEDIA_ACCELERATOR", "cpu").strip() or "cpu"
if accelerator not in ("cpu", "cuda", "rocm"):
raise ConfigError("MEDIA_ACCELERATOR must be cpu, cuda or rocm")
optional = frozenset(
item.strip() for item in env.get("MEDIA_OPTIONAL_BACKENDS", "").split(",") if item.strip()
)
return Settings(
token=token,
media_root=media_root,
models_dir=env.get("MEDIA_MODELS_DIR", "/models").strip() or "/models",
host=env.get("MEDIA_WORKER_HOST", "127.0.0.1").strip() or "127.0.0.1",
port=_int(env, "MEDIA_WORKER_PORT", 8790, 1),
accelerator=accelerator,
gpu_concurrency=_int(env, "MEDIA_GPU_CONCURRENCY", 1, 1),
cpu_concurrency=_int(env, "MEDIA_CPU_CONCURRENCY", 2, 1),
queue_size=_int(env, "MEDIA_QUEUE_SIZE", 8, 0),
max_input_bytes=_int(env, "MEDIA_MAX_INPUT_BYTES", 20 * GIB, 1),
max_seconds=_int(env, "MEDIA_MAX_SECONDS", 4 * 3600, 1),
allow_model_download=env.get("MEDIA_MODEL_DOWNLOAD", "1") != "0",
optional_backends=optional,
insecure_dev=insecure,
)

View File

@ -0,0 +1,41 @@
"""Hardware video encoders that actually encode a frame on this host."""
from __future__ import annotations
import functools
import os
import re
import subprocess
from .media_io import FFMPEG
CANDIDATE = re.compile(r"^\s*V\S*\s+((?:h264|hevc|av1)_(?:nvenc|vaapi|qsv|amf|videotoolbox|v4l2m2m))\s")
RENDER_NODE = "/dev/dri/renderD128"
def _works(name: str) -> bool:
argv = [FFMPEG, "-v", "error", "-nostdin"]
if name.endswith("_vaapi"):
if not os.path.exists(RENDER_NODE):
return False
argv += ["-vaapi_device", RENDER_NODE]
argv += ["-f", "lavfi", "-i", "color=c=black:s=256x256:d=0.1", "-frames:v", "1"]
if name.endswith("_vaapi"):
argv += ["-vf", "format=nv12,hwupload"]
argv += ["-c:v", name, "-f", "null", "-"]
try:
return subprocess.run(argv, capture_output=True, timeout=15, shell=False).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
@functools.cache
def hardware_encoders() -> tuple[str, ...]:
try:
listing = subprocess.run(
[FFMPEG, "-hide_banner", "-encoders"], capture_output=True, timeout=15, shell=False, text=True
).stdout
except (OSError, subprocess.TimeoutExpired):
return ()
names = [m.group(1) for line in listing.splitlines() if (m := CANDIDATE.match(line))]
return tuple(name for name in names if _works(name))

View File

@ -0,0 +1,62 @@
"""Error envelope shared by every route: {"error": {"code", "message", "operation"}}."""
from __future__ import annotations
from fastapi.responses import JSONResponse
STATUS_BY_CODE = {
"bad_request": 400,
"invalid_path": 400,
"unauthorized": 401,
"too_large": 413,
"unsupported": 501,
"busy": 503,
"internal": 500,
}
class MediaError(Exception):
def __init__(self, code: str, message: str, operation: str | None = None) -> None:
super().__init__(message)
self.code = code
self.message = message
self.operation = operation
@property
def status(self) -> int:
return STATUS_BY_CODE[self.code]
class Unsupported(MediaError):
def __init__(self, reason: str, operation: str | None = None) -> None:
super().__init__("unsupported", reason, operation)
class InvalidPath(MediaError):
def __init__(self, message: str = "path is outside MEDIA_ROOT") -> None:
super().__init__("invalid_path", message)
class BadRequest(MediaError):
def __init__(self, message: str) -> None:
super().__init__("bad_request", message)
class Busy(MediaError):
def __init__(self) -> None:
super().__init__("busy", "inference queue is full")
class Cancelled(Exception):
"""The client went away or the request ran past MEDIA_MAX_SECONDS."""
def __init__(self, reason: str = "client disconnected") -> None:
super().__init__(reason)
self.reason = reason
def error_response(code: str, message: str, operation: str | None, status: int | None = None) -> JSONResponse:
return JSONResponse(
status_code=status or STATUS_BY_CODE[code],
content={"error": {"code": code, "message": message, "operation": operation}},
)

View File

@ -0,0 +1,153 @@
"""HTTP API v1 of the Media Studio sidecar.
GET /healthz is open; everything else needs the bearer token. POST
/v1/<operation> takes the TypeScript request object (minus `signal`) and
answers with the TypeScript result object.
"""
from __future__ import annotations
import logging
import threading
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any
import anyio
from fastapi import Depends, FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, Response
from pydantic import BaseModel, ValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from . import VERSION, encoders, schemas
from .auth import require_token
from .config import Settings
from .errors import Cancelled, MediaError, error_response
from .models import OPERATIONS, Registry
from .ops import audio, embeddings, faces, inpaint, redact, saliency, scenes, segment, synthesize, transcribe
from .ops.common import Ctx, jsonable
from .ops.devices import effective_accelerator
from .runtime import Gate, run_job
log = logging.getLogger("media-worker")
MAX_BODY_BYTES = 64 * 1024 * 1024
# Status for a request whose client went away; nobody reads it.
CLIENT_CLOSED = 499
@dataclass(frozen=True)
class Route:
schema: type[BaseModel]
run: Callable[[Ctx, Any, threading.Event], dict[str, Any]]
gpu: bool
ROUTES: dict[str, Route] = {
"detectScenes": Route(schemas.PathRequest, scenes.run, False),
"transcribe": Route(schemas.TranscribeRequest, transcribe.run, True),
"detectFaces": Route(schemas.DetectFacesRequest, faces.detect_run, True),
"trackFaces": Route(schemas.TrackFacesRequest, faces.track_run, True),
"faceLandmarks": Route(schemas.TimesRequest, faces.landmarks_run, True),
"embedFrames": Route(schemas.TimesRequest, embeddings.embed_run, True),
"classifyContent": Route(schemas.ClassifyContentRequest, embeddings.classify_run, True),
"segmentPrompted": Route(schemas.SegmentPromptedRequest, segment.run, True),
"inpaintVideo": Route(schemas.InpaintVideoRequest, inpaint.run, True),
"redact": Route(schemas.RedactRequest, redact.run, False),
"synthesizeIdentity": Route(schemas.SynthesizeIdentityRequest, synthesize.run, True),
"audioFeatures": Route(schemas.PathRequest, audio.run, False),
"saliency": Route(schemas.TimesRequest, saliency.run, False),
}
assert set(ROUTES) == set(OPERATIONS)
def _operation_of(request: Request) -> str | None:
name = request.url.path.rsplit("/", 1)[-1]
return name if name in ROUTES else None
def _validation_message(error: ValidationError) -> str:
first = error.errors()[0] if error.errors() else {}
where = ".".join(str(part) for part in first.get("loc", ()))
return (
f"{where}: {first.get('msg', 'invalid request')}"
if where
else str(first.get("msg", "invalid request"))
)
def create_app(settings: Settings | None = None, registry: Registry | None = None) -> FastAPI:
settings = settings or Settings.from_env()
registry = registry or Registry(settings)
app = FastAPI(title="media-worker", version=VERSION, docs_url=None, redoc_url=None, openapi_url=None)
app.state.settings = settings
app.state.ctx = Ctx(settings, registry)
app.state.gpu_gate = Gate(settings.gpu_concurrency, settings.queue_size)
app.state.cpu_gate = Gate(settings.cpu_concurrency, settings.queue_size)
@app.exception_handler(MediaError)
async def media_error(request: Request, error: MediaError) -> JSONResponse:
return error_response(error.code, error.message, error.operation or _operation_of(request))
@app.exception_handler(RequestValidationError)
async def validation_error(request: Request, error: RequestValidationError) -> JSONResponse:
return error_response("bad_request", "invalid request", _operation_of(request))
@app.exception_handler(StarletteHTTPException)
async def http_error(request: Request, error: StarletteHTTPException) -> JSONResponse:
code = "bad_request" if error.status_code < 500 else "internal"
return error_response(code, str(error.detail), _operation_of(request), status=error.status_code)
@app.exception_handler(Exception)
async def unexpected(request: Request, error: Exception) -> JSONResponse:
log.exception("unhandled error")
return error_response("internal", "internal error", _operation_of(request))
@app.get("/healthz")
async def healthz() -> dict[str, Any]:
return {"ok": True, "version": VERSION}
@app.get("/v1/capabilities", dependencies=[Depends(require_token)])
async def capabilities() -> dict[str, Any]:
hardware = await anyio.to_thread.run_sync(encoders.hardware_encoders)
return {
"configured": True,
"operations": registry.operation_capabilities(),
"accelerator": effective_accelerator(settings),
"hardwareEncoders": list(hardware),
}
@app.post("/v1/{operation}", dependencies=[Depends(require_token)])
async def operate(operation: str, request: Request) -> Response:
route = ROUTES.get(operation)
if route is None:
return error_response("bad_request", "unknown operation", None, status=404)
raw = bytearray()
async for chunk in request.stream():
raw.extend(chunk)
if len(raw) > MAX_BODY_BYTES:
raise MediaError("too_large", "request body too large", operation)
try:
body = route.schema.model_validate_json(bytes(raw))
except ValidationError as error:
raise MediaError("bad_request", _validation_message(error), operation) from error
ctx: Ctx = request.app.state.ctx
gate: Gate = request.app.state.gpu_gate if route.gpu else request.app.state.cpu_gate
try:
result = await run_job(
request, gate, lambda cancel: route.run(ctx, body, cancel), settings.max_seconds
)
except MediaError as error:
error.operation = operation
raise
except Cancelled as error:
if error.reason == "client disconnected":
return Response(status_code=CLIENT_CLOSED)
raise MediaError("internal", error.reason, operation) from error
except Exception as error:
log.exception("%s failed", operation)
raise MediaError("internal", "internal error", operation) from error
return JSONResponse(jsonable(result))
return app

View File

@ -0,0 +1,435 @@
"""ffmpeg/ffprobe plumbing. Every call is an argv list; nothing goes through a shell."""
from __future__ import annotations
import contextlib
import json
import os
import queue
import re
import subprocess
import threading
import uuid
from collections import deque
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from fractions import Fraction
import numpy as np
from .errors import BadRequest, Cancelled, MediaError
FFMPEG = "ffmpeg"
FFPROBE = "ffprobe"
SHOWINFO_PTS = re.compile(rb"pts_time:\s*(-?[0-9.]+)")
@dataclass(frozen=True)
class VideoInfo:
width: int
height: int
fps: float
fps_text: str
duration_s: float
has_audio: bool
has_video: bool
start_s: float = 0.0
# Video stream duration when the container reports it (audio may run longer).
video_duration_s: float = 0.0
def frame_ms(self) -> float:
return 1000.0 / self.fps
def run_cancellable(
argv: list[str], cancel: threading.Event, timeout: float | None = None
) -> subprocess.CompletedProcess[bytes]:
"""Run a short-lived tool, killing it when the request is cancelled."""
proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
killer = _kill_on_cancel(proc, cancel)
try:
stdout, stderr = proc.communicate(timeout=timeout)
finally:
killer.set()
if cancel.is_set():
raise Cancelled()
return subprocess.CompletedProcess(argv, proc.returncode, stdout, stderr)
def _kill_on_cancel(proc: subprocess.Popen[bytes], cancel: threading.Event) -> threading.Event:
done = threading.Event()
def watch() -> None:
while not done.is_set():
if cancel.wait(0.2):
if proc.poll() is None:
proc.kill()
return
threading.Thread(target=watch, daemon=True).start()
return done
def _parse_rate(text: str | None) -> float:
if not text or text in ("0/0", "0"):
return 0.0
try:
return float(Fraction(text))
except (ValueError, ZeroDivisionError):
return 0.0
def probe(path: str, cancel: threading.Event) -> VideoInfo:
result = run_cancellable(
[
FFPROBE,
"-v",
"error",
"-print_format",
"json",
"-show_streams",
"-show_format",
path,
],
cancel,
)
if result.returncode != 0:
raise BadRequest("input is not a readable media file")
data = json.loads(result.stdout or b"{}")
streams = data.get("streams", [])
video = next(
(
s
for s in streams
if s.get("codec_type") == "video" and not s.get("disposition", {}).get("attached_pic")
),
None,
)
audio = next((s for s in streams if s.get("codec_type") == "audio"), None)
duration = float(data.get("format", {}).get("duration") or 0.0)
if video is None:
return VideoInfo(0, 0, 0.0, "0", duration, audio is not None, False)
fps_text = video.get("avg_frame_rate") or "0/0"
fps = _parse_rate(fps_text)
if fps <= 0 or fps > 240:
fps_text = video.get("r_frame_rate") or "0/0"
fps = _parse_rate(fps_text)
if fps <= 0:
fps_text, fps = "25", 25.0
width, height = int(video.get("width") or 0), int(video.get("height") or 0)
rotation = _rotation(video)
if rotation in (90, 270):
width, height = height, width
if not duration:
duration = float(video.get("duration") or 0.0)
start = float(video.get("start_time") or 0.0)
video_duration = float(video.get("duration") or 0.0) or duration
return VideoInfo(width, height, fps, fps_text, duration, audio is not None, True, start, video_duration)
def _rotation(stream: dict) -> int:
for side in stream.get("side_data_list", []) or []:
if "rotation" in side:
return int(abs(int(side["rotation"]))) % 360
tag = (stream.get("tags") or {}).get("rotate")
return int(abs(int(tag))) % 360 if tag else 0
def check_duration(info: VideoInfo, max_seconds: int) -> None:
if info.duration_s > max_seconds:
raise MediaError("too_large", "input duration exceeds MEDIA_MAX_SECONDS")
def scaled_size(width: int, height: int, max_side: int | None) -> tuple[int, int]:
if not max_side or max(width, height) <= max_side:
return width - width % 2, height - height % 2
scale = max_side / max(width, height)
w = max(2, int(round(width * scale / 2)) * 2)
h = max(2, int(round(height * scale / 2)) * 2)
return w, h
class _RawReader:
"""Read fixed-size BGR frames from an ffmpeg stdout pipe."""
def __init__(
self,
argv: list[str],
width: int,
height: int,
cancel: threading.Event,
pts: queue.Queue[float] | None = None,
) -> None:
self.size = width * height * 3
self.shape = (height, width, 3)
self.proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
self.cancel = cancel
self.pts = pts
self._stderr: deque[bytes] = deque(maxlen=20)
self._drain = threading.Thread(target=self._read_stderr, daemon=True)
self._drain.start()
def _read_stderr(self) -> None:
assert self.proc.stderr is not None
for line in self.proc.stderr:
if self.pts is not None and b"Parsed_showinfo" in line:
match = SHOWINFO_PTS.search(line)
if match:
self.pts.put(float(match.group(1)))
continue
self._stderr.append(line)
def frames(self) -> Iterator[np.ndarray]:
assert self.proc.stdout is not None
try:
while True:
if self.cancel.is_set():
raise Cancelled()
buf = self.proc.stdout.read(self.size)
if not buf:
break
if len(buf) != self.size:
raise MediaError("internal", "truncated frame from decoder")
yield np.frombuffer(buf, np.uint8).reshape(self.shape)
code = self.proc.wait()
self._drain.join(timeout=5)
if code != 0:
raise MediaError("internal", "decoder failed: " + self.stderr_tail())
finally:
self.close()
def stderr_tail(self) -> str:
return b"".join(self._stderr).decode("utf-8", "replace").strip()[-500:]
def close(self) -> None:
if self.proc.poll() is None:
self.proc.kill()
self.proc.wait()
def stream_frames(
path: str,
info: VideoInfo,
cancel: threading.Event,
sample_fps: float | None = None,
max_side: int | None = None,
) -> Iterator[tuple[int, np.ndarray]]:
"""Yield (tMs, frame) for every frame (CFR-normalized) or about `sample_fps` per second.
Sampled frames are real source frames; their timestamps come from
ffmpeg's showinfo, so tMs is exact even for variable frame rates.
"""
width, height = scaled_size(info.width, info.height, max_side)
filters: list[str] = []
if sample_fps:
# The source frame nearest to each k / sample_fps slot (phase-locked, no drift).
half = 0.5 / info.fps
slot = f"floor((t+{half:.6f})*{sample_fps:.6f})"
prev = f"floor((prev_selected_t+{half:.6f})*{sample_fps:.6f})"
filters += [f"select='isnan(prev_selected_t)+gt({slot}\\,{prev})'", "showinfo"]
if (width, height) != (info.width, info.height):
filters.append(f"scale={width}:{height}:flags=area")
level = "info" if sample_fps else "error"
argv = [FFMPEG, "-v", level, "-nostats", "-nostdin", "-i", path, "-map", "0:v:0", "-an", "-sn"]
if filters:
argv += ["-vf", ",".join(filters)]
if sample_fps:
argv += ["-fps_mode", "passthrough"]
else:
argv += ["-fps_mode", "cfr", "-r", info.fps_text]
argv += ["-f", "rawvideo", "-pix_fmt", "bgr24", "pipe:1"]
pts: queue.Queue[float] | None = queue.Queue() if sample_fps else None
reader = _RawReader(argv, width, height, cancel, pts)
for index, frame in enumerate(reader.frames()):
if pts is None:
yield int(round(index * 1000.0 / info.fps)), frame
continue
try:
t = pts.get(timeout=30)
except queue.Empty as error:
raise MediaError("internal", "missing frame timestamp from decoder") from error
yield max(int(round((t - info.start_s) * 1000)), 0), frame
def frame_at(
path: str, info: VideoInfo, t_ms: int, cancel: threading.Event, max_side: int | None = None
) -> np.ndarray:
"""Decode the frame shown at t_ms (accurate seek)."""
width, height = scaled_size(info.width, info.height, max_side)
t = min(max(t_ms / 1000.0, 0.0), max(info.duration_s - 0.5 / info.fps, 0.0))
argv = [
FFMPEG,
"-v",
"error",
"-nostdin",
"-ss",
f"{t:.3f}",
"-i",
path,
"-map",
"0:v:0",
"-frames:v",
"1",
]
if (width, height) != (info.width, info.height):
argv += ["-vf", f"scale={width}:{height}:flags=area"]
argv += ["-f", "rawvideo", "-pix_fmt", "bgr24", "pipe:1"]
frames = list(_RawReader(argv, width, height, cancel).frames())
if not frames:
raise MediaError("internal", f"no frame decoded at {t_ms} ms")
return frames[0]
def frames_at(
path: str, info: VideoInfo, times_ms: list[int], cancel: threading.Event, max_side: int | None = None
) -> Iterator[tuple[int, np.ndarray]]:
for t_ms in times_ms:
if cancel.is_set():
raise Cancelled()
yield t_ms, frame_at(path, info, t_ms, cancel, max_side)
def temp_sibling(final_path: str) -> str:
directory, name = os.path.split(final_path)
stem, ext = os.path.splitext(name)
return os.path.join(directory, f".{stem}.partial-{uuid.uuid4().hex}{ext}")
def encoder_args(out_path: str) -> list[str]:
"""Near-lossless intermediate: FFV1 in Matroska, otherwise x264 CRF 12."""
if out_path.lower().endswith(".mkv"):
return ["-c:v", "ffv1", "-level", "3", "-pix_fmt", "yuv444p", "-g", "1"]
return ["-c:v", "libx264", "-preset", "medium", "-crf", "12", "-pix_fmt", "yuv420p"]
FrameFn = Callable[[int, int, np.ndarray], np.ndarray]
def transform_video(
path: str,
out_path: str,
info: VideoInfo,
cancel: threading.Event,
frame_fn: FrameFn,
) -> int:
"""Decode every frame, apply frame_fn(index, tMs, frame), encode, copy audio.
Fails closed: any decode, transform, or encode error removes the temporary
output and raises; out_path only ever appears complete (atomic rename).
"""
if not info.has_video:
raise BadRequest("input has no video stream")
width, height = scaled_size(info.width, info.height, None)
tmp = temp_sibling(out_path)
enc_argv = [
FFMPEG,
"-v",
"error",
"-nostdin",
"-y",
"-f",
"rawvideo",
"-pix_fmt",
"bgr24",
"-s",
f"{width}x{height}",
"-r",
info.fps_text,
"-i",
"pipe:0",
"-i",
path,
"-map",
"0:v:0",
"-map",
"1:a?",
"-map_metadata",
"-1",
"-map_chapters",
"-1",
*encoder_args(out_path),
"-c:a",
"copy",
tmp,
]
dec_argv = [
FFMPEG,
"-v",
"error",
"-xerror",
"-nostdin",
"-i",
path,
"-map",
"0:v:0",
"-an",
"-sn",
"-fps_mode",
"cfr",
"-r",
info.fps_text,
]
if (width, height) != (info.width, info.height):
dec_argv += ["-vf", f"crop={width}:{height}:0:0"]
dec_argv += ["-f", "rawvideo", "-pix_fmt", "bgr24", "pipe:1"]
encoder = subprocess.Popen(
enc_argv, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, shell=False
)
enc_err: list[bytes] = []
drain = threading.Thread(target=lambda: enc_err.extend(encoder.stderr or []), daemon=True)
drain.start()
reader = _RawReader(dec_argv, width, height, cancel)
count = 0
ok = False
try:
for index, frame in enumerate(reader.frames()):
t_ms = int(round(index * 1000.0 / info.fps))
out = frame_fn(index, t_ms, frame.copy())
if out.shape != frame.shape or out.dtype != np.uint8:
raise MediaError("internal", "frame transform changed the frame shape")
assert encoder.stdin is not None
try:
encoder.stdin.write(np.ascontiguousarray(out).tobytes())
except BrokenPipeError as error:
raise MediaError("internal", "encoder stopped accepting frames") from error
count += 1
expected = (info.video_duration_s or info.duration_s) * info.fps
if count == 0 or (expected > 0 and count < expected * 0.98 - 2):
raise MediaError("internal", f"decoded {count} of ~{int(expected)} frames")
assert encoder.stdin is not None
encoder.stdin.close()
if encoder.wait() != 0:
drain.join(timeout=5)
raise MediaError(
"internal", "encoder failed: " + b"".join(enc_err[-5:]).decode("utf-8", "replace")[-500:]
)
drain.join(timeout=5)
os.replace(tmp, out_path)
ok = True
return count
finally:
reader.close()
if not ok:
if encoder.poll() is None:
encoder.kill()
with contextlib.suppress(subprocess.TimeoutExpired):
encoder.wait(timeout=10)
if os.path.exists(tmp):
os.unlink(tmp)
_NUMBER = r"(-?(?:inf|\d+(?:\.\d+)?))"
def parse_ebur128(stderr: str) -> tuple[float | None, float | None]:
"""Integrated loudness and true peak from the ebur128 summary block."""
summary = stderr[stderr.rfind("Summary:") :] if "Summary:" in stderr else ""
lufs = re.search(r"I:\s+" + _NUMBER + r"\s+LUFS", summary)
peak = re.search(r"Peak:\s+" + _NUMBER + r"\s+dBFS", summary)
def num(match: re.Match[str] | None) -> float | None:
if not match or "inf" in match.group(1):
return None
return float(match.group(1))
return num(lufs), num(peak)

View File

@ -0,0 +1,506 @@
"""Model registry: which backend serves each operation, its license, and its files.
Weights are never part of the image or the repo. They download lazily on first
use into MEDIA_MODELS_DIR/<backend id>/ and are verified against the pinned
SHA256 and size before anything loads them. A mismatch deletes the file and
the operation reports `unsupported` with the reason.
Default backends are commercially usable. Anything else must be listed in
MEDIA_OPTIONAL_BACKENDS by the operator before it runs.
"""
from __future__ import annotations
import hashlib
import importlib.util
import logging
import os
import threading
import urllib.request
from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any
from .errors import Unsupported
log = logging.getLogger("media-worker.models")
OPERATIONS = (
"detectScenes",
"transcribe",
"detectFaces",
"trackFaces",
"faceLandmarks",
"embedFrames",
"classifyContent",
"segmentPrompted",
"inpaintVideo",
"redact",
"synthesizeIdentity",
"audioFeatures",
"saliency",
)
HF = "https://huggingface.co"
YUNET_REV = "3cc26e7f1014a5ee5d74a42acee58bafc9d0a310"
SFACE_REV = "3d7082438a6e4551e840c9b2bb60b71e8da4b524"
WHISPER_REV = "536b0662742c02347bc0e980a01041f333bce120"
SIGLIP_REV = "4649052661e53c7000355844105f8a1792088239"
LAMA_REV = "c3c0c9e468934d62e79c329e35d82dd09ff8c444"
SAM2_REV = "de431c4043854a71d8101e17995dfe596bf101a5"
@dataclass(frozen=True)
class ModelFile:
name: str
url: str
sha256: str
sizeBytes: int
@dataclass(frozen=True)
class BackendSpec:
"""One way to serve an operation. `id` doubles as the model cache directory."""
id: str
model: str | None
license: str | None
commercialUse: bool | None
experimental: bool
source: str | None
files: tuple[ModelFile, ...] = ()
# Python modules that must be importable.
requires: tuple[str, ...] = ()
# Set when the backend can never run (e.g. no licensed model exists).
unavailable_reason: str | None = None
@dataclass
class Backend:
spec: BackendSpec
# Builds the runtime object from {file name: verified local path} and the settings.
load: Callable[[dict[str, str], Any], Any] | None = None
def _yunet_file() -> ModelFile:
return ModelFile(
"face_detection_yunet_2023mar.onnx",
f"{HF}/opencv/face_detection_yunet/resolve/{YUNET_REV}/face_detection_yunet_2023mar.onnx",
"8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4",
232589,
)
def _siglip(name: str, sha256: str, size: int) -> ModelFile:
path = name if name == "tokenizer.json" else f"onnx/{name}"
return ModelFile(name, f"{HF}/Xenova/siglip-base-patch16-224/resolve/{SIGLIP_REV}/{path}", sha256, size)
def _whisper(name: str, sha256: str, size: int) -> ModelFile:
return ModelFile(name, f"{HF}/Systran/faster-whisper-small/resolve/{WHISPER_REV}/{name}", sha256, size)
def _lazy(module: str, attr: str) -> Callable[[dict[str, str], Any], Any]:
def load(files: dict[str, str], settings: Any) -> Any:
mod = importlib.import_module(f"app.ops.{module}")
return getattr(mod, attr)(files, settings)
return load
YUNET = BackendSpec(
id="yunet-2023mar",
model="opencv-yunet-2023mar",
license="MIT",
commercialUse=True,
experimental=False,
source="https://github.com/opencv/opencv_zoo/tree/main/models/face_detection_yunet",
files=(_yunet_file(),),
requires=("cv2",),
)
SIGLIP_SOURCE = "https://huggingface.co/google/siglip-base-patch16-224"
SIGLIP_VISION = _siglip(
"vision_model_quantized.onnx",
"ef14a954f3d57e1806666432bd9785004c1dc27100aa260eee0cb0f10a5de058",
99499129,
)
SIGLIP_FULL = _siglip(
"model_quantized.onnx",
"9171eb00c38b9ec82f924877356d008b79e3285dbac7cd10965827bee30c9a99",
210977441,
)
SIGLIP_TOKENIZER = _siglip(
"tokenizer.json",
"4a17c975210be5ab4c36b47d8dae4eefb866dbfb1e676e394aad85dc30a3ae08",
2398744,
)
def default_backends() -> dict[str, Backend]:
return {
"detectScenes": Backend(
BackendSpec(
id="pyscenedetect-adaptive",
model="pyscenedetect-adaptive",
license="BSD-3-Clause",
commercialUse=True,
experimental=False,
source="https://github.com/Breakthrough/PySceneDetect",
requires=("scenedetect", "cv2"),
),
_lazy("scenes", "load_scene_detector"),
),
"transcribe": Backend(
BackendSpec(
id="faster-whisper-small",
model="faster-whisper-small",
license="MIT",
commercialUse=True,
experimental=False,
source="https://huggingface.co/Systran/faster-whisper-small",
files=(
_whisper(
"config.json",
"b55496ac7940a7ae47d2c01eab40edfd8701feec1229d9cce3b40014383fb828",
2370,
),
_whisper(
"model.bin",
"3e305921506d8872816023e4c273e75d2419fb89b24da97b4fe7bce14170d671",
483546902,
),
_whisper(
"tokenizer.json",
"fb7b63191e9bb045082c79fd742a3106a12c99513ab30df4a0d47fa6cb6fd0ab",
2203239,
),
_whisper(
"vocabulary.txt",
"34ce3fe1c5041027b3f8d42912270993f986dbc4bb34cf27f951e34a1e453913",
459861,
),
),
requires=("faster_whisper",),
),
_lazy("transcribe", "load_transcriber"),
),
"detectFaces": Backend(YUNET, _lazy("faces", "load_face_detector")),
"trackFaces": Backend(
BackendSpec(
id=YUNET.id,
model="opencv-yunet-2023mar+iou-tracker",
license="MIT",
commercialUse=True,
experimental=False,
source=YUNET.source,
files=YUNET.files,
requires=YUNET.requires,
),
_lazy("faces", "load_face_detector"),
),
"faceLandmarks": Backend(
BackendSpec(
id=YUNET.id,
model="opencv-yunet-2023mar-5pt",
license="MIT",
commercialUse=True,
experimental=False,
source=YUNET.source,
files=YUNET.files,
requires=YUNET.requires,
),
_lazy("faces", "load_face_detector"),
),
"embedFrames": Backend(
BackendSpec(
id="siglip-base-patch16-224",
model="siglip-base-patch16-224",
license="Apache-2.0",
commercialUse=True,
experimental=False,
source=SIGLIP_SOURCE,
files=(SIGLIP_VISION,),
requires=("onnxruntime",),
),
_lazy("embeddings", "load_frame_embedder"),
),
"classifyContent": Backend(
BackendSpec(
id="siglip-base-patch16-224",
model="siglip-base-patch16-224-zero-shot",
license="Apache-2.0",
commercialUse=True,
# Zero-shot prompts, not a trained moderation head.
experimental=True,
source=SIGLIP_SOURCE,
files=(SIGLIP_FULL, SIGLIP_TOKENIZER),
requires=("onnxruntime", "tokenizers"),
),
_lazy("embeddings", "load_content_classifier"),
),
"segmentPrompted": Backend(
BackendSpec(
id="sam2.1-hiera-tiny",
model="sam2.1-hiera-tiny",
license="Apache-2.0",
commercialUse=True,
experimental=True,
source="https://github.com/facebookresearch/sam2",
files=(
ModelFile(
"sam2.1_hiera_tiny.pt",
f"{HF}/facebook/sam2.1-hiera-tiny/resolve/{SAM2_REV}/sam2.1_hiera_tiny.pt",
"7402e0d864fa82708a20fbd15bc84245c2f26dff0eb43a4b5b93452deb34be69",
156008466,
),
),
requires=("torch", "sam2"),
),
_lazy("segment", "load_segmenter"),
),
"inpaintVideo": Backend(
BackendSpec(
id="lama-big-onnx",
model="big-lama+dis-flow",
license="Apache-2.0",
commercialUse=True,
experimental=True,
source="https://huggingface.co/Carve/LaMa-ONNX",
files=(
ModelFile(
"lama_fp32.onnx",
f"{HF}/Carve/LaMa-ONNX/resolve/{LAMA_REV}/lama_fp32.onnx",
"1faef5301d78db7dda502fe59966957ec4b79dd64e16f03ed96913c7a4eb68d6",
208044816,
),
),
requires=("onnxruntime", "cv2"),
),
_lazy("inpaint", "load_inpainter"),
),
"redact": Backend(
BackendSpec(
id="opencv-redact",
model="opencv-redact",
license="Apache-2.0",
commercialUse=True,
experimental=False,
source="https://github.com/opencv/opencv",
requires=("cv2",),
),
),
"synthesizeIdentity": Backend(
BackendSpec(
id="synthetic-identity",
model=None,
license=None,
commercialUse=None,
experimental=True,
source=None,
unavailable_reason="no commercially licensed model",
),
),
"audioFeatures": Backend(
BackendSpec(
id="ffmpeg-ebur128-librosa",
model="ffmpeg-ebur128+librosa-beat",
license="ISC",
commercialUse=True,
experimental=False,
source="https://github.com/librosa/librosa",
requires=("librosa",),
),
),
"saliency": Backend(
BackendSpec(
id="spectral-residual",
model="spectral-residual",
license="Apache-2.0",
commercialUse=True,
experimental=False,
source="https://doi.org/10.1109/CVPR.2007.383267",
requires=("cv2",),
),
),
}
# Opt-in backends that are not clean enough to be defaults.
SFACE = Backend(
BackendSpec(
id="sface-2021dec",
model="opencv-sface-2021dec",
license="Apache-2.0 (weights); training data provenance unverified",
commercialUse=None,
experimental=True,
source="https://github.com/opencv/opencv_zoo/tree/main/models/face_recognition_sface",
files=(
ModelFile(
"face_recognition_sface_2021dec.onnx",
f"{HF}/opencv/face_recognition_sface/resolve/{SFACE_REV}/face_recognition_sface_2021dec.onnx",
"0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79",
38696353,
),
),
requires=("cv2",),
),
_lazy("faces", "load_identity_embedder"),
)
OPTIONAL_BACKENDS = {"faceIdentity": SFACE}
def _sha256(path: str) -> str:
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def _module_available(name: str) -> bool:
try:
return importlib.util.find_spec(name) is not None
except (ImportError, ValueError):
return False
@dataclass
class Registry:
settings: Any
backends: dict[str, Backend] = field(default_factory=default_backends)
optional: dict[str, Backend] = field(default_factory=lambda: dict(OPTIONAL_BACKENDS))
_runtimes: dict[str, Any] = field(default_factory=dict)
_verified: set[str] = field(default_factory=set)
_failed: dict[str, str] = field(default_factory=dict)
_lock: threading.Lock = field(default_factory=threading.Lock)
_file_locks: dict[str, threading.Lock] = field(default_factory=dict)
# -- availability -------------------------------------------------------
def _license_block(self, spec: BackendSpec, key: str) -> str | None:
if spec.commercialUse is True:
return None
if key in self.settings.optional_backends or spec.id in self.settings.optional_backends:
return None
return "backend is not licensed for commercial use and not enabled in MEDIA_OPTIONAL_BACKENDS"
def unavailable_reason(self, backend: Backend, key: str) -> str | None:
spec = backend.spec
if spec.unavailable_reason:
return spec.unavailable_reason
blocked = self._license_block(spec, key)
if blocked:
return blocked
missing = [name for name in spec.requires if not _module_available(name)]
if missing:
return f"python module not installed: {', '.join(missing)}"
if spec.id in self._failed:
return self._failed[spec.id]
if not self.settings.allow_model_download:
absent = [f.name for f in spec.files if not os.path.isfile(self._file_path(spec, f))]
if absent:
return "model files not installed and MEDIA_MODEL_DOWNLOAD=0"
return None
def backend(self, operation: str) -> Backend:
return self.backends[operation]
def optional_enabled(self, key: str) -> bool:
backend = self.optional.get(key)
return backend is not None and self.unavailable_reason(backend, key) is None
# -- runtime -------------------------------------------------------------
def runtime(self, operation: str) -> Any:
"""Return the loaded runtime for an operation, downloading weights on first use."""
return self._runtime(self.backends[operation], operation)
def optional_runtime(self, key: str) -> Any:
return self._runtime(self.optional[key], key)
def _runtime(self, backend: Backend, key: str) -> Any:
reason = self.unavailable_reason(backend, key)
if reason:
raise Unsupported(reason)
cache_key = f"{backend.spec.id}:{key}"
with self._lock:
if cache_key in self._runtimes:
return self._runtimes[cache_key]
files = {f.name: self.ensure_file(backend.spec, f) for f in backend.spec.files}
runtime = backend.load(files, self.settings) if backend.load else None
with self._lock:
self._runtimes.setdefault(cache_key, runtime)
return self._runtimes[cache_key]
# -- files ---------------------------------------------------------------
def _file_path(self, spec: BackendSpec, model_file: ModelFile) -> str:
return os.path.join(self.settings.models_dir, spec.id, model_file.name)
def _file_lock(self, path: str) -> threading.Lock:
with self._lock:
return self._file_locks.setdefault(path, threading.Lock())
def ensure_file(self, spec: BackendSpec, model_file: ModelFile) -> str:
path = self._file_path(spec, model_file)
with self._file_lock(path):
if path in self._verified:
return path
if not os.path.isfile(path):
if not self.settings.allow_model_download:
raise Unsupported(f"model file {model_file.name} is not installed")
self._download(model_file, path)
ok = os.path.getsize(path) == model_file.sizeBytes and _sha256(path) == model_file.sha256
if not ok:
os.unlink(path)
reason = f"model file {model_file.name} failed SHA256 verification"
self._failed[spec.id] = reason
raise Unsupported(reason)
self._verified.add(path)
return path
def _download(self, model_file: ModelFile, path: str) -> None:
os.makedirs(os.path.dirname(path), exist_ok=True)
partial = f"{path}.part"
log.info("downloading %s", model_file.url)
try:
with (
urllib.request.urlopen(model_file.url, timeout=60) as response,
open(partial, "wb") as handle,
):
written = 0
for chunk in iter(lambda: response.read(1 << 20), b""):
written += len(chunk)
if written > model_file.sizeBytes:
raise ValueError("download larger than pinned size")
handle.write(chunk)
os.replace(partial, path)
except Exception as error:
if os.path.exists(partial):
os.unlink(partial)
raise Unsupported(f"model file {model_file.name} could not be downloaded: {error}") from error
# -- capabilities ----------------------------------------------------------
def operation_capabilities(self) -> dict[str, dict[str, Any]]:
result: dict[str, dict[str, Any]] = {}
for operation in OPERATIONS:
backend = self.backends[operation]
spec = backend.spec
result[operation] = {
"available": self.unavailable_reason(backend, operation) is None,
"experimental": spec.experimental,
"model": spec.model,
"license": spec.license,
"commercialUse": spec.commercialUse,
}
return result
def prefetch(self) -> None:
"""Download and verify every default backend's files (operator warm-up)."""
for operation in OPERATIONS:
backend = self.backends[operation]
if self.unavailable_reason(backend, operation) is None:
for model_file in backend.spec.files:
self.ensure_file(backend.spec, model_file)

View File

@ -0,0 +1 @@
"""One module per operation family. Each exposes `run(ctx, body, cancel)`."""

View File

@ -0,0 +1,163 @@
"""audioFeatures without ML: ffmpeg ebur128 + silencedetect, librosa beat tracking.
Audio is decoded once to mono 22.05 kHz float32 and processed in five-minute
chunks, so memory stays bounded and the request can stop between chunks.
"""
from __future__ import annotations
import re
import subprocess
import threading
from typing import Any
import numpy as np
from ..errors import Cancelled, MediaError
from ..media_io import FFMPEG, parse_ebur128, run_cancellable
from ..schemas import PathRequest
from .common import Ctx, open_input
SAMPLE_RATE = 22050
CHUNK_SECONDS = 300
ENERGY_HOP_MS = 250
SILENCE_DB = -40
SILENCE_MIN_S = 0.5
def loudness_and_silence(
path: str, cancel: threading.Event
) -> tuple[float | None, float | None, list[dict[str, int]]]:
result = run_cancellable(
[
FFMPEG,
"-hide_banner",
"-nostats",
"-nostdin",
"-i",
path,
"-map",
"0:a:0",
"-vn",
"-af",
f"ebur128=peak=true,silencedetect=noise={SILENCE_DB}dB:d={SILENCE_MIN_S}",
"-f",
"null",
"-",
],
cancel,
)
stderr = result.stderr.decode("utf-8", "replace")
if result.returncode != 0:
raise MediaError("internal", "loudness measurement failed")
lufs, peak = parse_ebur128(stderr)
silences: list[dict[str, int]] = []
start: float | None = None
for line in stderr.splitlines():
if (m := re.search(r"silence_start:\s*(-?[\d.]+)", line)) is not None:
start = max(float(m.group(1)), 0.0)
elif (m := re.search(r"silence_end:\s*([\d.]+)", line)) is not None and start is not None:
silences.append(
{"startMs": int(round(start * 1000)), "endMs": int(round(float(m.group(1)) * 1000))}
)
start = None
if start is not None:
silences.append({"startMs": int(round(start * 1000)), "endMs": -1})
return lufs, peak, silences
def decode_chunks(path: str, cancel: threading.Event) -> Any:
argv = [
FFMPEG,
"-v",
"error",
"-nostdin",
"-i",
path,
"-map",
"0:a:0",
"-vn",
"-ac",
"1",
"-ar",
str(SAMPLE_RATE),
"-f",
"f32le",
"pipe:1",
]
proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, shell=False)
chunk_bytes = SAMPLE_RATE * CHUNK_SECONDS * 4
assert proc.stdout is not None
try:
while True:
if cancel.is_set():
raise Cancelled()
buf = proc.stdout.read(chunk_bytes)
if not buf:
break
usable = len(buf) - len(buf) % 4
yield np.frombuffer(buf[:usable], dtype="<f4").astype(np.float32)
if proc.wait() != 0:
raise MediaError("internal", "audio decode failed")
finally:
if proc.poll() is None:
proc.kill()
proc.wait()
def run(ctx: Ctx, body: PathRequest, cancel: threading.Event) -> dict[str, Any]:
import librosa # deferred: heavy import (numba), only needed here
real, info = open_input(ctx, body.path, cancel, need_video=False)
ctx.registry.runtime("audioFeatures")
if not info.has_audio:
return {
"integratedLufs": None,
"truePeakDb": None,
"tempoBpm": None,
"beatsMs": [],
"energy": [],
"silences": [],
}
lufs, peak, silences = loudness_and_silence(real, cancel)
hop = int(SAMPLE_RATE * ENERGY_HOP_MS / 1000)
beats: list[int] = []
tempos: list[tuple[float, float]] = []
rms_values: list[float] = []
offset_s = 0.0
for samples in decode_chunks(real, cancel):
length_s = samples.size / SAMPLE_RATE
if samples.size >= SAMPLE_RATE * 2 and float(np.abs(samples).max()) > 1e-4:
tempo, frames = librosa.beat.beat_track(y=samples, sr=SAMPLE_RATE, units="frames")
times = librosa.frames_to_time(frames, sr=SAMPLE_RATE)
beats.extend(int(round((offset_s + t) * 1000)) for t in times)
value = float(np.atleast_1d(tempo)[0])
if value > 0:
tempos.append((value, length_s))
for i in range(0, samples.size, hop):
window = samples[i : i + hop]
rms_values.append(float(np.sqrt(np.mean(window.astype(np.float64) ** 2))) if window.size else 0.0)
offset_s += length_s
duration_ms = int(round(offset_s * 1000))
for silence in silences:
if silence["endMs"] < 0:
silence["endMs"] = duration_ms
peak_rms = max(rms_values) if rms_values else 0.0
energy = [
{"tMs": i * ENERGY_HOP_MS, "value": (v / peak_rms) if peak_rms > 0 else 0.0}
for i, v in enumerate(rms_values)
]
tempo_bpm = None
if tempos:
weights = np.array([w for _, w in tempos])
tempo_bpm = float(np.average([t for t, _ in tempos], weights=weights))
return {
"integratedLufs": lufs,
"truePeakDb": peak,
"tempoBpm": tempo_bpm,
"beatsMs": sorted(set(beats)),
"energy": energy,
"silences": silences,
}

View File

@ -0,0 +1,62 @@
"""Helpers shared by the operation modules."""
from __future__ import annotations
import threading
from dataclasses import dataclass
from typing import Any
import numpy as np
from .. import media_io, paths
from ..config import Settings
from ..errors import BadRequest
from ..media_io import VideoInfo
from ..models import Registry
@dataclass
class Ctx:
settings: Settings
registry: Registry
def open_input(
ctx: Ctx, path: str, cancel: threading.Event, need_video: bool = True
) -> tuple[str, VideoInfo]:
"""Resolve, size-check, probe, and duration-check an input file."""
real = paths.input_file(ctx.settings.media_root, path, ctx.settings.max_input_bytes)
info = media_io.probe(real, cancel)
media_io.check_duration(info, ctx.settings.max_seconds)
if need_video and not info.has_video:
raise BadRequest("input has no video stream")
return real, info
def clamp01(value: float) -> float:
return float(min(max(value, 0.0), 1.0))
def norm_box(x: float, y: float, w: float, h: float, width: int, height: int) -> dict[str, float]:
"""Pixel box -> normalized box clipped to the frame."""
x0, y0 = clamp01(x / width), clamp01(y / height)
x1, y1 = clamp01((x + w) / width), clamp01((y + h) / height)
return {"x": x0, "y": y0, "w": max(x1 - x0, 0.0), "h": max(y1 - y0, 0.0)}
def norm_point(x: float, y: float, width: int, height: int) -> dict[str, float]:
return {"x": clamp01(x / width), "y": clamp01(y / height)}
def unique_sorted(times: list[int]) -> list[int]:
return sorted(set(int(t) for t in times))
def jsonable(value: Any) -> Any:
if isinstance(value, dict):
return {k: jsonable(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [jsonable(v) for v in value]
if isinstance(value, np.generic):
return value.item()
return value

View File

@ -0,0 +1,61 @@
"""Accelerator detection. ML libraries are imported only when installed."""
from __future__ import annotations
import functools
import importlib.util
from typing import Any
def _installed(name: str) -> bool:
try:
return importlib.util.find_spec(name) is not None
except (ImportError, ValueError):
return False
@functools.cache
def torch_gpu_available() -> bool:
if not _installed("torch"):
return False
import torch
return bool(torch.cuda.is_available())
def effective_accelerator(settings: Any) -> str:
"""What the image was built for, downgraded to cpu when no GPU is visible.
ROCm builds of torch expose the GPU through the torch.cuda API.
"""
if settings.accelerator in ("cuda", "rocm") and torch_gpu_available():
return settings.accelerator
return "cpu"
def torch_device(settings: Any) -> str:
return "cuda" if effective_accelerator(settings) != "cpu" else "cpu"
def ctranslate2_device(settings: Any) -> str:
# CTranslate2 ships CUDA builds only; ROCm hosts transcribe on the CPU.
if settings.accelerator != "cuda":
return "cpu"
import ctranslate2
return "cuda" if ctranslate2.get_cuda_device_count() > 0 else "cpu"
def onnx_providers(settings: Any) -> list[str]:
import onnxruntime
available = set(onnxruntime.get_available_providers())
providers: list[str] = []
if settings.accelerator == "cuda" and "CUDAExecutionProvider" in available:
providers.append("CUDAExecutionProvider")
if settings.accelerator == "rocm":
for name in ("MIGraphXExecutionProvider", "ROCMExecutionProvider"):
if name in available:
providers.append(name)
providers.append("CPUExecutionProvider")
return providers

View File

@ -0,0 +1,147 @@
"""embedFrames and classifyContent with SigLIP (Apache-2.0) on ONNX Runtime.
classifyContent is zero-shot: each category is a set of text prompts and the
score is the highest sigmoid probability among them. It is reported as
experimental; the gate thresholds decide what the scores mean.
"""
from __future__ import annotations
import threading
from typing import Any
import cv2
import numpy as np
from .. import media_io
from ..schemas import ClassifyContentRequest, TimesRequest
from .common import Ctx, clamp01, open_input, unique_sorted
from .devices import onnx_providers
SIGLIP_SIZE = 224
SIGLIP_TEXT_LENGTH = 64
SIGLIP_PAD_ID = 1
BATCH = 16
CATEGORY_PROMPTS: dict[str, tuple[str, ...]] = {
"explicit_nudity": ("a photo of a naked person", "a photo containing explicit nudity"),
"suggestive": ("a photo of a person in lingerie", "a sexually suggestive photo"),
"violence": ("a photo of a violent fight", "a photo showing blood and injuries"),
"minors_risk": ("a photo of a child", "a photo of a young teenager"),
"self_harm": ("a photo of self-harm", "a photo of cuts on a wrist"),
"drugs": ("a photo of illegal drugs", "a photo of drug paraphernalia"),
"weapons": ("a photo of a gun", "a photo of a person holding a knife as a weapon"),
}
def preprocess(frames: list[np.ndarray]) -> np.ndarray:
"""BGR frames -> SigLIP pixel_values (N, 3, 224, 224), normalized to [-1, 1]."""
batch = []
for frame in frames:
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
resized = cv2.resize(rgb, (SIGLIP_SIZE, SIGLIP_SIZE), interpolation=cv2.INTER_CUBIC)
batch.append(resized.astype(np.float32) / 127.5 - 1.0)
return np.ascontiguousarray(np.stack(batch).transpose(0, 3, 1, 2))
def _session(path: str, settings: Any) -> Any:
import onnxruntime
return onnxruntime.InferenceSession(path, providers=onnx_providers(settings))
class SiglipEmbedder:
model = "siglip-base-patch16-224"
def __init__(self, path: str, settings: Any) -> None:
self.session = _session(path, settings)
names = [o.name for o in self.session.get_outputs()]
self.output = "pooler_output" if "pooler_output" in names else names[-1]
self.dim = 768
def embed(self, frames: list[np.ndarray]) -> np.ndarray:
(vectors,) = self.session.run([self.output], {"pixel_values": preprocess(frames)})
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
return vectors / np.maximum(norms, 1e-12)
def load_frame_embedder(files: dict[str, str], settings: Any) -> SiglipEmbedder:
return SiglipEmbedder(files["vision_model_quantized.onnx"], settings)
class SiglipZeroShot:
model = "siglip-base-patch16-224-zero-shot"
def __init__(self, model_path: str, tokenizer_path: str, settings: Any) -> None:
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file(tokenizer_path)
tokenizer.enable_truncation(SIGLIP_TEXT_LENGTH)
tokenizer.enable_padding(length=SIGLIP_TEXT_LENGTH, pad_id=SIGLIP_PAD_ID)
self.labels: list[str] = []
prompts: list[str] = []
for category, texts in CATEGORY_PROMPTS.items():
for text in texts:
self.labels.append(category)
prompts.append(text.lower())
self.input_ids = np.array([e.ids for e in tokenizer.encode_batch(prompts)], dtype=np.int64)
self.session = _session(model_path, settings)
def classify(self, frames: list[np.ndarray]) -> list[dict[str, float]]:
(logits,) = self.session.run(
["logits_per_image"], {"input_ids": self.input_ids, "pixel_values": preprocess(frames)}
)
probs = 1.0 / (1.0 + np.exp(-logits))
result = []
for row in probs:
scores: dict[str, float] = {}
for label, value in zip(self.labels, row, strict=True):
scores[label] = max(scores.get(label, 0.0), clamp01(float(value)))
result.append(scores)
return result
def load_content_classifier(files: dict[str, str], settings: Any) -> SiglipZeroShot:
return SiglipZeroShot(files["model_quantized.onnx"], files["tokenizer.json"], settings)
def _batched(frames: Any, size: int) -> Any:
times: list[int] = []
batch: list[np.ndarray] = []
for t_ms, frame in frames:
times.append(t_ms)
batch.append(frame)
if len(batch) == size:
yield times, batch
times, batch = [], []
if batch:
yield times, batch
def embed_run(ctx: Ctx, body: TimesRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
embedder = ctx.registry.runtime("embedFrames")
vectors: list[list[float]] = []
# Vectors follow the request order, duplicates included.
for _, batch in _batched(media_io.frames_at(real, info, list(body.timesMs), cancel, 512), BATCH):
vectors.extend([float(v) for v in row] for row in embedder.embed(batch))
return {"model": embedder.model, "dim": int(embedder.dim), "vectors": vectors}
def classify_run(ctx: Ctx, body: ClassifyContentRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
classifier = ctx.registry.runtime("classifyContent")
results: dict[int, dict[str, float]] = {}
def score(frames: Any) -> None:
for times, batch in _batched(frames, BATCH):
for t_ms, scores in zip(times, classifier.classify(batch), strict=True):
results[t_ms] = scores
score(media_io.stream_frames(real, info, cancel, body.sampleFps, 512))
extra = [t for t in unique_sorted(body.extraTimesMs or []) if t not in results]
score(media_io.frames_at(real, info, extra, cancel, 512))
return {
"model": classifier.model,
"frames": [{"tMs": t, "scores": results[t]} for t in sorted(results)],
}

View File

@ -0,0 +1,265 @@
"""detectFaces, trackFaces, faceLandmarks.
Detection is OpenCV YuNet (MIT). Tracking is a ByteTrack-style IoU tracker
implemented here: high-confidence detections associate first, low-confidence
ones only extend existing tracks, and a track ends after 500 ms without a match.
Identity embeddings (for local person clustering) come from an opt-in backend
and are null otherwise.
"""
from __future__ import annotations
import math
import threading
from dataclasses import dataclass, field
from typing import Any
import cv2
import numpy as np
from .. import media_io
from ..schemas import DetectFacesRequest, TimesRequest, TrackFacesRequest
from .common import Ctx, clamp01, norm_box, norm_point, open_input, unique_sorted
DETECT_MAX_SIDE = 1280
DETECT_THRESHOLD = 0.6
TRACK_LOW_THRESHOLD = 0.3
TRACK_MATCH_IOU = 0.3
TRACK_MAX_GAP_MS = 500
@dataclass
class Detection:
"""Pixel-space face: box (x, y, w, h), score, five landmarks (x, y)."""
box: tuple[float, float, float, float]
score: float
landmarks: list[tuple[float, float]]
class YuNetDetector:
def __init__(self, model_path: str) -> None:
self.model_path = model_path
self._local = threading.local()
def _detector(self, width: int, height: int) -> Any:
detector = getattr(self._local, "detector", None)
if detector is None:
detector = cv2.FaceDetectorYN.create(
self.model_path, "", (width, height), TRACK_LOW_THRESHOLD, 0.3, 5000
)
self._local.detector = detector
detector.setInputSize((width, height))
return detector
def detect(self, frame: np.ndarray) -> list[Detection]:
height, width = frame.shape[:2]
_, faces = self._detector(width, height).detect(frame)
result: list[Detection] = []
for row in faces if faces is not None else []:
points = [(float(row[4 + 2 * i]), float(row[5 + 2 * i])) for i in range(5)]
result.append(
Detection(
(float(row[0]), float(row[1]), float(row[2]), float(row[3])), float(row[14]), points
)
)
return result
def load_face_detector(files: dict[str, str], _settings: Any) -> YuNetDetector:
return YuNetDetector(files["face_detection_yunet_2023mar.onnx"])
class SFaceEmbedder:
def __init__(self, model_path: str) -> None:
self.model_path = model_path
self._local = threading.local()
def embed(self, frame: np.ndarray, detection: Detection) -> np.ndarray:
recognizer = getattr(self._local, "recognizer", None)
if recognizer is None:
recognizer = cv2.FaceRecognizerSF.create(self.model_path, "")
self._local.recognizer = recognizer
row = np.array(
[*detection.box, *[c for p in detection.landmarks for c in p], detection.score],
dtype=np.float32,
)
aligned = recognizer.alignCrop(frame, row)
return recognizer.feature(aligned).reshape(-1).astype(np.float32)
def load_identity_embedder(files: dict[str, str], _settings: Any) -> SFaceEmbedder:
return SFaceEmbedder(files["face_recognition_sface_2021dec.onnx"])
def _observation(det: Detection, width: int, height: int) -> dict[str, Any]:
return {
"box": norm_box(*det.box, width, height),
"confidence": clamp01(det.score),
"landmarks": [norm_point(x, y, width, height) for x, y in det.landmarks],
}
def detect_run(ctx: Ctx, body: DetectFacesRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
detector = ctx.registry.runtime("detectFaces")
frames = (
media_io.frames_at(real, info, unique_sorted(body.timesMs), cancel, DETECT_MAX_SIDE)
if body.timesMs is not None
else media_io.stream_frames(real, info, cancel, body.sampleFps, DETECT_MAX_SIDE)
)
out = []
for t_ms, frame in frames:
height, width = frame.shape[:2]
faces = [
_observation(d, width, height) for d in detector.detect(frame) if d.score >= DETECT_THRESHOLD
]
out.append({"tMs": t_ms, "faces": faces})
return {"frames": out}
def landmarks_run(ctx: Ctx, body: TimesRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
detector = ctx.registry.runtime("faceLandmarks")
out = []
for t_ms, frame in media_io.frames_at(real, info, unique_sorted(body.timesMs), cancel, DETECT_MAX_SIDE):
height, width = frame.shape[:2]
faces = [
{
"box": norm_box(*d.box, width, height),
"landmarks": [norm_point(x, y, width, height) for x, y in d.landmarks],
}
for d in detector.detect(frame)
if d.score >= DETECT_THRESHOLD
]
out.append({"tMs": t_ms, "faces": faces})
return {"frames": out}
def iou(a: tuple[float, float, float, float], b: tuple[float, float, float, float]) -> float:
ax1, ay1, ax2, ay2 = a[0], a[1], a[0] + a[2], a[1] + a[3]
bx1, by1, bx2, by2 = b[0], b[1], b[0] + b[2], b[1] + b[3]
iw = max(0.0, min(ax2, bx2) - max(ax1, bx1))
ih = max(0.0, min(ay2, by2) - max(ay1, by1))
inter = iw * ih
union = a[2] * a[3] + b[2] * b[3] - inter
return inter / union if union > 0 else 0.0
@dataclass
class Track:
track_id: str
box: tuple[float, float, float, float]
velocity: tuple[float, float] = (0.0, 0.0)
last_ms: int = 0
keyframes: list[dict[str, Any]] = field(default_factory=list)
# Best (score, frame crop source, detection) for identity embeddings.
best: list[tuple[float, np.ndarray, Detection]] = field(default_factory=list)
def predicted(self, t_ms: int) -> tuple[float, float, float, float]:
dt = t_ms - self.last_ms
return (
self.box[0] + self.velocity[0] * dt,
self.box[1] + self.velocity[1] * dt,
self.box[2],
self.box[3],
)
class IouTracker:
"""Two-stage greedy IoU association (ByteTrack's idea without the Kalman filter)."""
def __init__(self, max_gap_ms: int = TRACK_MAX_GAP_MS) -> None:
self.max_gap_ms = max_gap_ms
self.active: list[Track] = []
self.finished: list[Track] = []
self._next = 1
def _associate(
self, tracks: list[Track], dets: list[Detection], t_ms: int
) -> tuple[list[tuple[Track, Detection]], list[Track], list[Detection]]:
pairs = sorted(
(
(iou(tr.predicted(t_ms), d.box), ti, di)
for ti, tr in enumerate(tracks)
for di, d in enumerate(dets)
),
reverse=True,
)
used_t: set[int] = set()
used_d: set[int] = set()
matches: list[tuple[Track, Detection]] = []
for score, ti, di in pairs:
if score < TRACK_MATCH_IOU:
break
if ti in used_t or di in used_d:
continue
used_t.add(ti)
used_d.add(di)
matches.append((tracks[ti], dets[di]))
rest_t = [t for i, t in enumerate(tracks) if i not in used_t]
rest_d = [d for i, d in enumerate(dets) if i not in used_d]
return matches, rest_t, rest_d
def update(
self, t_ms: int, dets: list[Detection], frame: np.ndarray | None, keep_crops: bool
) -> list[tuple[Track, Detection]]:
expired = [t for t in self.active if t_ms - t.last_ms > self.max_gap_ms]
self.finished.extend(expired)
self.active = [t for t in self.active if t not in expired]
high = [d for d in dets if d.score >= DETECT_THRESHOLD]
low = [d for d in dets if TRACK_LOW_THRESHOLD <= d.score < DETECT_THRESHOLD]
matches, rest_tracks, rest_high = self._associate(self.active, high, t_ms)
low_matches, _, _ = self._associate(rest_tracks, low, t_ms)
matches += low_matches
for track, det in matches:
dt = max(t_ms - track.last_ms, 1)
track.velocity = ((det.box[0] - track.box[0]) / dt, (det.box[1] - track.box[1]) / dt)
track.box = det.box
track.last_ms = t_ms
for det in rest_high:
track = Track(f"t{self._next}", det.box, last_ms=t_ms)
self._next += 1
self.active.append(track)
matches.append((track, det))
if keep_crops and frame is not None:
for track, det in matches:
if det.score >= DETECT_THRESHOLD:
track.best.append((det.score, frame, det))
track.best.sort(key=lambda item: -item[0])
del track.best[3:]
return matches
def close(self) -> list[Track]:
tracks = self.finished + self.active
self.active, self.finished = [], []
return sorted(tracks, key=lambda t: int(t.track_id[1:]))
def track_run(ctx: Ctx, body: TrackFacesRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
detector = ctx.registry.runtime("trackFaces")
embedder = (
ctx.registry.optional_runtime("faceIdentity")
if ctx.registry.optional_enabled("faceIdentity")
else None
)
tracker = IouTracker()
for t_ms, frame in media_io.stream_frames(real, info, cancel, body.sampleFps, DETECT_MAX_SIDE):
height, width = frame.shape[:2]
matches = tracker.update(t_ms, detector.detect(frame), frame, embedder is not None)
for track, det in matches:
track.keyframes.append(
{"tMs": t_ms, "box": norm_box(*det.box, width, height), "confidence": clamp01(det.score)}
)
tracks = []
for track in tracker.close():
embedding = None
if embedder is not None and track.best:
vectors = np.stack([embedder.embed(frame, det) for _, frame, det in track.best])
mean = vectors.mean(axis=0)
norm = float(np.linalg.norm(mean))
if norm > 0 and math.isfinite(norm):
embedding = [float(v) for v in mean / norm]
tracks.append({"trackId": track.track_id, "keyframes": track.keyframes, "embedding": embedding})
return {"sampleFps": body.sampleFps, "tracks": tracks}

View File

@ -0,0 +1,163 @@
"""inpaintVideo: LaMa (Apache-2.0, ONNX) per frame with optical-flow smoothing.
Per frame: crop around the mask, inpaint at 512x512, paste back through a
feathered mask. When the previous frame was inpainted too, its result is
warped into the current frame with DIS optical flow and blended in, which
damps flicker. Quality is measured on the output:
- temporalStability: 1 - mean absolute change inside the mask between
consecutive frames after motion compensation (scaled so 32 levels -> 0).
- edgeQuality: 1 - excess gradient on the mask boundary over the gradient
just outside it.
- score: 0.6 * temporalStability + 0.4 * edgeQuality.
Callers fall back to cover/reframe when the score is low.
"""
from __future__ import annotations
import os
import threading
from typing import Any
import cv2
import numpy as np
from .. import media_io, paths
from ..errors import BadRequest
from ..schemas import InpaintVideoRequest
from .common import Ctx, clamp01, open_input
from .devices import onnx_providers
from .masks import MaskIndex
from .redact import OUTPUT_EXTENSIONS
LAMA_SIZE = 512
TEMPORAL_BLEND = 0.35
class LamaInpainter:
def __init__(self, path: str, settings: Any) -> None:
import onnxruntime
self.session = onnxruntime.InferenceSession(path, providers=onnx_providers(settings))
def inpaint(self, image: np.ndarray, mask: np.ndarray) -> np.ndarray:
"""image: BGR uint8 512x512, mask: uint8 0/255 512x512 -> BGR uint8."""
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
pixel = rgb.transpose(2, 0, 1)[None]
m = (mask > 127).astype(np.float32)[None, None]
(out,) = self.session.run(["output"], {"image": pixel, "mask": m})
result = np.clip(out[0].transpose(1, 2, 0), 0, 255).astype(np.uint8)
return cv2.cvtColor(result, cv2.COLOR_RGB2BGR)
def load_inpainter(files: dict[str, str], settings: Any) -> LamaInpainter:
return LamaInpainter(files["lama_fp32.onnx"], settings)
def _crop_box(mask: np.ndarray) -> tuple[int, int, int, int]:
height, width = mask.shape
ys, xs = np.nonzero(mask)
cx, cy = (xs.min() + xs.max()) / 2, (ys.min() + ys.max()) / 2
side = max(xs.max() - xs.min() + 1, ys.max() - ys.min() + 1) * 2.2
side = int(min(max(side, 256), max(width, height)))
x0 = int(min(max(cx - side / 2, 0), max(width - side, 0)))
y0 = int(min(max(cy - side / 2, 0), max(height - side, 0)))
return x0, y0, min(x0 + side, width), min(y0 + side, height)
class Smoother:
"""Temporal blending and quality bookkeeping across frames."""
def __init__(self) -> None:
self.flow = cv2.DISOpticalFlow_create(cv2.DISOPTICAL_FLOW_PRESET_FAST)
self.prev_gray: np.ndarray | None = None
self.prev_out: np.ndarray | None = None
self.prev_masked = False
self.stability: list[float] = []
self.edges: list[float] = []
def _warp_prev(self, gray: np.ndarray) -> np.ndarray | None:
if self.prev_gray is None or self.prev_out is None:
return None
flow = self.flow.calc(gray, self.prev_gray, None)
h, w = gray.shape
grid_x, grid_y = np.meshgrid(np.arange(w, dtype=np.float32), np.arange(h, dtype=np.float32))
return cv2.remap(
self.prev_out,
grid_x + flow[..., 0],
grid_y + flow[..., 1],
cv2.INTER_LINEAR,
borderMode=cv2.BORDER_REPLICATE,
)
def step(self, source: np.ndarray, inpainted: np.ndarray | None, mask: np.ndarray | None) -> np.ndarray:
gray = cv2.cvtColor(source, cv2.COLOR_BGR2GRAY)
out = source
if inpainted is not None and mask is not None:
soft = cv2.GaussianBlur(mask.astype(np.float32) / 255.0, (0, 0), 3)
soft = np.maximum(soft, mask.astype(np.float32) / 255.0)[..., None]
warped = self._warp_prev(gray) if self.prev_masked else None
fill = inpainted.astype(np.float32)
if warped is not None:
fill = fill * (1 - TEMPORAL_BLEND) + warped.astype(np.float32) * TEMPORAL_BLEND
out = np.clip(source.astype(np.float32) * (1 - soft) + fill * soft, 0, 255).astype(np.uint8)
inside = mask > 0
if warped is not None:
# Raw LaMa output against the motion-compensated previous frame: flicker before smoothing.
diff = np.abs(inpainted.astype(np.float32) - warped.astype(np.float32))[inside].mean()
self.stability.append(clamp01(1.0 - float(diff) / 32.0))
self.edges.append(_edge_quality(out, mask))
self.prev_gray = gray
self.prev_out = out
self.prev_masked = mask is not None
return out
def quality(self) -> dict[str, float]:
temporal = float(np.mean(self.stability)) if self.stability else 1.0
edge = float(np.mean(self.edges)) if self.edges else 1.0
return {
"temporalStability": clamp01(temporal),
"edgeQuality": clamp01(edge),
"score": clamp01(0.6 * temporal + 0.4 * edge),
}
def _edge_quality(frame: np.ndarray, mask: np.ndarray) -> float:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
grad = cv2.magnitude(cv2.Sobel(gray, cv2.CV_32F, 1, 0), cv2.Sobel(gray, cv2.CV_32F, 0, 1))
kernel = np.ones((5, 5), np.uint8)
boundary = (cv2.dilate(mask, kernel) > 0) & (cv2.erode(mask, kernel) == 0)
outer = (cv2.dilate(mask, np.ones((15, 15), np.uint8)) > 0) & ~(cv2.dilate(mask, kernel) > 0)
if not boundary.any() or not outer.any():
return 1.0
excess = max(0.0, float(grad[boundary].mean()) - float(grad[outer].mean()))
return clamp01(1.0 - excess / (float(grad[outer].mean()) + 40.0))
def run(ctx: Ctx, body: InpaintVideoRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
masks = MaskIndex(paths.input_dir(ctx.settings.media_root, body.maskDir), info.frame_ms())
out_real = paths.output_file(ctx.settings.media_root, body.outPath)
if os.path.splitext(out_real)[1].lower() not in OUTPUT_EXTENSIONS:
raise BadRequest("outPath must end in .mkv, .mp4 or .mov")
if out_real == real:
raise BadRequest("outPath must differ from path")
inpainter = ctx.registry.runtime("inpaintVideo")
smoother = Smoother()
def per_frame(_index: int, t_ms: int, frame: np.ndarray) -> np.ndarray:
height, width = frame.shape[:2]
mask = masks.lookup(t_ms, width, height)
if mask is None or not mask.any():
return smoother.step(frame, None, None)
# Grow slightly so the seam lands on untouched pixels.
mask = cv2.dilate(mask, np.ones((7, 7), np.uint8))
x0, y0, x1, y1 = _crop_box(mask)
crop = cv2.resize(frame[y0:y1, x0:x1], (LAMA_SIZE, LAMA_SIZE), interpolation=cv2.INTER_AREA)
crop_mask = cv2.resize(mask[y0:y1, x0:x1], (LAMA_SIZE, LAMA_SIZE), interpolation=cv2.INTER_NEAREST)
filled = inpainter.inpaint(crop, crop_mask)
full = frame.copy()
full[y0:y1, x0:x1] = cv2.resize(filled, (x1 - x0, y1 - y0), interpolation=cv2.INTER_CUBIC)
return smoother.step(frame, full, mask)
media_io.transform_video(real, out_real, info, cancel, per_frame)
return {"outPath": body.outPath, "quality": smoother.quality()}

View File

@ -0,0 +1,54 @@
"""Per-frame mask directories (`<tMs>.png`), as written by segmentPrompted."""
from __future__ import annotations
import bisect
import os
import re
import cv2
import numpy as np
from ..errors import BadRequest, MediaError
MASK_NAME = re.compile(r"^(\d+)\.png$")
class MaskIndex:
def __init__(self, directory: str, frame_ms: float) -> None:
entries = []
for name in os.listdir(directory):
match = MASK_NAME.match(name)
if match:
entries.append((int(match.group(1)), os.path.join(directory, name)))
if not entries:
raise BadRequest("mask directory contains no <tMs>.png masks")
entries.sort()
self.times = [t for t, _ in entries]
self.files = [p for _, p in entries]
gaps = np.diff(self.times) if len(self.times) > 1 else np.array([frame_ms])
# A mask covers frames up to 1.5 sampling intervals away (at least one frame).
self.tolerance = max(float(np.median(gaps)) * 1.5, frame_ms)
self._cache: tuple[int, np.ndarray] | None = None
def lookup(self, t_ms: int, width: int, height: int) -> np.ndarray | None:
"""Binary mask (uint8 0/255) for the frame at t_ms, resized to the frame, or None."""
i = bisect.bisect_left(self.times, t_ms)
best = None
for j in (i - 1, i):
if not 0 <= j < len(self.times) or abs(self.times[j] - t_ms) > self.tolerance:
continue
if best is None or abs(self.times[j] - t_ms) < abs(self.times[best] - t_ms):
best = j
if best is None:
return None
if self._cache is not None and self._cache[0] == best:
return self._cache[1]
mask = cv2.imread(self.files[best], cv2.IMREAD_GRAYSCALE)
if mask is None:
raise MediaError("internal", f"unreadable mask {os.path.basename(self.files[best])}")
if mask.shape != (height, width):
mask = cv2.resize(mask, (width, height), interpolation=cv2.INTER_NEAREST)
mask = np.where(mask > 127, 255, 0).astype(np.uint8)
self._cache = (best, mask)
return mask

View File

@ -0,0 +1,271 @@
"""redact: frame-accurate face and region redaction (OpenCV + ffmpeg pipes).
Geometry per instruction and frame:
- Boxes interpolate linearly between keyframes. A single keyframe is a static
region for the whole video; otherwise the region applies from the first to
the last keyframe (plus half a frame either side).
- `paddingPct` grows the box by that percentage of its width/height (half on
each side). Mask regions are dilated by the same share of their extent.
- `featherPct` fades the effect outward over that percentage of the region's
shorter side. The region itself is always fully covered.
Fails closed: an unreadable frame, a mask that cannot be read, or an encoder
error is a 500 and no file is left at outPath.
"""
from __future__ import annotations
import bisect
import os
import threading
from dataclasses import dataclass
from typing import Any
import cv2
import numpy as np
from .. import media_io, paths
from ..errors import BadRequest
from ..schemas import RedactionInstruction, RedactRequest
from .common import Ctx, open_input
from .masks import MaskIndex
OUTPUT_EXTENSIONS = (".mkv", ".mp4", ".mov")
DEFAULT_FILL = (0, 0, 0)
def hex_to_bgr(color: str | None) -> tuple[int, int, int] | None:
if not color:
return None
value = color.lstrip("#")
r, g, b = int(value[0:2], 16), int(value[2:4], 16), int(value[4:6], 16)
return (b, g, r)
@dataclass
class Plan:
mode: str
padding: float
feather: float
color: tuple[int, int, int] | None
times: list[int]
boxes: list[tuple[float, float, float, float]]
landmark_times: list[int]
landmarks: list[list[tuple[float, float]]]
masks: MaskIndex | None
def build_plan(ctx: Ctx, instruction: RedactionInstruction, frame_ms: float) -> Plan:
keyframes = sorted(instruction.keyframes, key=lambda k: k.tMs)
landmarks = sorted(instruction.landmarks or [], key=lambda frame: frame.tMs)
masks = None
if instruction.maskDir:
masks = MaskIndex(paths.input_dir(ctx.settings.media_root, instruction.maskDir), frame_ms)
return Plan(
mode=instruction.mode,
padding=instruction.paddingPct / 100.0,
feather=instruction.featherPct / 100.0,
color=hex_to_bgr(instruction.color),
times=[k.tMs for k in keyframes],
boxes=[(k.box.x, k.box.y, k.box.w, k.box.h) for k in keyframes],
landmark_times=[frame.tMs for frame in landmarks],
landmarks=[[(p.x, p.y) for p in frame.points] for frame in landmarks],
masks=masks,
)
def interpolate(times: list[int], values: list[Any], t_ms: float, frame_ms: float, hold_single: bool) -> Any:
"""Linear interpolation of tuples/lists over keyframe times; None outside the span."""
if not times:
return None
if len(times) == 1:
if hold_single or abs(t_ms - times[0]) <= frame_ms / 2:
return values[0]
return None
if t_ms < times[0] - frame_ms / 2 or t_ms > times[-1] + frame_ms / 2:
return None
if t_ms <= times[0]:
return values[0]
if t_ms >= times[-1]:
return values[-1]
i = bisect.bisect_right(times, t_ms)
t0, t1 = times[i - 1], times[i]
a, b = values[i - 1], values[i]
if t1 == t0:
return b
f = (t_ms - t0) / (t1 - t0)
if isinstance(a, list):
if len(a) != len(b):
return a if f < 0.5 else b
return [tuple(pa[k] + (pb[k] - pa[k]) * f for k in range(2)) for pa, pb in zip(a, b, strict=True)]
return tuple(a[k] + (b[k] - a[k]) * f for k in range(len(a)))
def _pixel_box(
box: tuple[float, float, float, float], padding: float, width: int, height: int
) -> tuple[int, int, int, int]:
x, y, w, h = box
x -= w * padding / 2
y -= h * padding / 2
w *= 1 + padding
h *= 1 + padding
x0 = int(np.floor(max(0.0, x) * width))
y0 = int(np.floor(max(0.0, y) * height))
x1 = int(np.ceil(min(1.0, x + w) * width))
y1 = int(np.ceil(min(1.0, y + h) * height))
return x0, y0, max(x1, x0), max(y1, y0)
def blur_effect(roi: np.ndarray, size: int) -> np.ndarray:
"""Strong, size-scaled blur: downsample, upsample, Gaussian."""
h, w = roi.shape[:2]
factor = max(4, size // 6)
small = cv2.resize(roi, (max(1, w // factor), max(1, h // factor)), interpolation=cv2.INTER_AREA)
up = cv2.resize(small, (w, h), interpolation=cv2.INTER_LINEAR)
k = max(3, (size // 3) | 1)
return cv2.GaussianBlur(up, (k, k), 0)
def pixelate_effect(roi: np.ndarray, size: int) -> np.ndarray:
h, w = roi.shape[:2]
block = max(4, size // 10)
small = cv2.resize(roi, (max(1, -(-w // block)), max(1, -(-h // block))), interpolation=cv2.INTER_AREA)
return cv2.resize(small, (w, h), interpolation=cv2.INTER_NEAREST)
def _ring_color(frame: np.ndarray, region: np.ndarray, margin: int) -> tuple[int, int, int]:
"""Median colour just outside the region: the 'skin tone patch' cover fill."""
kernel = np.ones((2 * margin + 1, 2 * margin + 1), np.uint8)
ring = (cv2.dilate(region, kernel) > 0) & (region == 0)
if not ring.any():
return DEFAULT_FILL
b, g, r = np.median(frame[ring], axis=0)
return (int(b), int(g), int(r))
def _region_for(
plan: Plan, t_ms: int, frame_ms: float, width: int, height: int
) -> tuple[np.ndarray, int] | None:
"""Binary region mask (uint8 0/255) and its characteristic size in px."""
region = np.zeros((height, width), np.uint8)
size = 0
hold = plan.masks is None
box = interpolate(plan.times, plan.boxes, t_ms, frame_ms, hold_single=hold)
if box is not None:
x0, y0, x1, y1 = _pixel_box(box, plan.padding, width, height)
raw_w, raw_h = box[2] * width, box[3] * height
if plan.mode in ("partial_eyes", "partial_upper", "partial_lower"):
_partial_region(plan, region, box, (x0, y0, x1, y1), t_ms, frame_ms, width, height)
else:
region[y0:y1, x0:x1] = 255
size = max(size, int(min(raw_w, raw_h)))
if plan.masks is not None:
mask = plan.masks.lookup(t_ms, width, height)
if mask is not None:
ys, xs = np.nonzero(mask)
if xs.size:
extent = int(min(xs.max() - xs.min() + 1, ys.max() - ys.min() + 1))
grow = int(round(extent * plan.padding / 2))
if grow > 0:
mask = cv2.dilate(mask, np.ones((2 * grow + 1, 2 * grow + 1), np.uint8))
region |= mask
size = max(size, extent)
if not region.any():
return None
return region, max(size, 1)
def _partial_region(
plan: Plan,
region: np.ndarray,
box: tuple[float, float, float, float],
padded: tuple[int, int, int, int],
t_ms: int,
frame_ms: float,
width: int,
height: int,
) -> None:
x0, y0, x1, y1 = padded
bx, by, bw, bh = box
points = interpolate(plan.landmark_times, plan.landmarks, t_ms, frame_ms, hold_single=False)
# Five-point landmarks: right eye, left eye, nose, mouth corners.
eyes = points[:2] if points and len(points) >= 2 else None
nose_y = points[2][1] if points and len(points) >= 3 else by + bh * 0.55
if plan.mode == "partial_eyes":
if eyes:
(ex1, ey1), (ex2, ey2) = eyes
else:
ex1, ey1, ex2, ey2 = bx + bw * 0.3, by + bh * 0.4, bx + bw * 0.7, by + bh * 0.4
cx, cy = (ex1 + ex2) / 2 * width, (ey1 + ey2) / 2 * height
angle = float(np.degrees(np.arctan2((ey2 - ey1) * height, (ex2 - ex1) * width)))
band_w = (x1 - x0) * 1.0
band_h = max(bh * height * 0.32 * (1 + plan.padding), 2.0)
rect = cv2.boxPoints(((cx, cy), (band_w, band_h), angle)).astype(np.int32)
cv2.fillConvexPoly(region, rect, 255)
elif plan.mode == "partial_upper":
split = int(round(min(max(nose_y, by), by + bh) * height))
region[y0 : max(split, y0), x0:x1] = 255
else:
split = int(round(min(max(nose_y, by), by + bh) * height))
region[min(split, y1) : y1, x0:x1] = 255
def apply_plan(frame: np.ndarray, plan: Plan, t_ms: int, frame_ms: float) -> None:
height, width = frame.shape[:2]
found = _region_for(plan, t_ms, frame_ms, width, height)
if found is None:
return
region, size = found
radius = int(round(plan.feather * size))
ys, xs = np.nonzero(region)
margin = radius + 2
x0, x1 = max(0, int(xs.min()) - margin), min(width, int(xs.max()) + margin + 1)
y0, y1 = max(0, int(ys.min()) - margin), min(height, int(ys.max()) + margin + 1)
roi = frame[y0:y1, x0:x1]
roi_region = region[y0:y1, x0:x1]
mode = plan.mode
if mode in ("blur", "partial_upper", "partial_lower"):
effected = blur_effect(roi, size)
elif mode == "pixelate":
effected = pixelate_effect(roi, size)
else:
if mode == "cover" and plan.color is None:
fill = _ring_color(frame, region, max(4, size // 20))
else:
fill = plan.color or DEFAULT_FILL
effected = np.empty_like(roi)
effected[:] = fill
alpha = roi_region.astype(np.float32) / 255.0
if radius > 0:
kernel = np.ones((2 * radius + 1, 2 * radius + 1), np.uint8)
grown = cv2.dilate(roi_region, kernel).astype(np.float32) / 255.0
k = 2 * radius + 1
soft = cv2.GaussianBlur(grown, (k, k), 0)
alpha = np.maximum(alpha, soft)
alpha = alpha[..., None]
blended = roi.astype(np.float32) * (1.0 - alpha) + effected.astype(np.float32) * alpha
roi[:] = np.clip(np.rint(blended), 0, 255).astype(np.uint8)
def apply_redactions(frame: np.ndarray, t_ms: int, plans: list[Plan], frame_ms: float) -> np.ndarray:
for plan in plans:
apply_plan(frame, plan, t_ms, frame_ms)
return frame
def run(ctx: Ctx, body: RedactRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
ctx.registry.runtime("redact")
out_real = paths.output_file(ctx.settings.media_root, body.outPath)
if os.path.splitext(out_real)[1].lower() not in OUTPUT_EXTENSIONS:
raise BadRequest("outPath must end in .mkv, .mp4 or .mov")
if out_real == real:
raise BadRequest("outPath must differ from path")
frame_ms = info.frame_ms()
plans = [build_plan(ctx, r, frame_ms) for r in body.redactions]
def per_frame(_index: int, t_ms: int, frame: np.ndarray) -> np.ndarray:
return apply_redactions(frame, t_ms, plans, frame_ms)
count = media_io.transform_video(real, out_real, info, cancel, per_frame)
return {"outPath": body.outPath, "framesProcessed": count}

View File

@ -0,0 +1,58 @@
"""saliency: spectral residual (Hou & Zhang, 2007), implemented with numpy/OpenCV.
The point is the saliency-weighted centroid of the top 5 % most salient
pixels; strength is how much those pixels stand out against the frame mean.
"""
from __future__ import annotations
import threading
from typing import Any
import cv2
import numpy as np
from .. import media_io
from ..schemas import TimesRequest
from .common import Ctx, clamp01, open_input, unique_sorted
WORK_SIZE = 64
def saliency_map(frame: np.ndarray) -> np.ndarray:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
small = cv2.resize(gray, (WORK_SIZE, WORK_SIZE), interpolation=cv2.INTER_AREA)
spectrum = np.fft.fft2(small)
log_amp = np.log(np.abs(spectrum) + 1e-8)
phase = np.angle(spectrum)
residual = log_amp - cv2.blur(log_amp, (3, 3))
sal = np.abs(np.fft.ifft2(np.exp(residual + 1j * phase))) ** 2
sal = cv2.GaussianBlur(sal.astype(np.float32), (0, 0), 2.5)
peak = float(sal.max())
return sal / peak if peak > 0 else sal
def salient_point(frame: np.ndarray) -> tuple[float, float, float]:
sal = saliency_map(frame)
threshold = float(np.quantile(sal, 0.95))
top = np.where(sal >= threshold, sal, 0.0)
total = float(top.sum())
if total <= 0:
return 0.5, 0.5, 0.0
ys, xs = np.mgrid[0 : sal.shape[0], 0 : sal.shape[1]]
cx = float((xs * top).sum() / total + 0.5) / sal.shape[1]
cy = float((ys * top).sum() / total + 0.5) / sal.shape[0]
mean = float(sal.mean())
top_mean = float(sal[sal >= threshold].mean())
strength = (top_mean - mean) / top_mean if top_mean > 0 else 0.0
return clamp01(cx), clamp01(cy), clamp01(strength)
def run(ctx: Ctx, body: TimesRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
ctx.registry.runtime("saliency")
points = []
for t_ms, frame in media_io.frames_at(real, info, unique_sorted(body.timesMs), cancel, 512):
cx, cy, strength = salient_point(frame)
points.append({"tMs": t_ms, "cx": cx, "cy": cy, "strength": strength})
return {"points": points}

View File

@ -0,0 +1,103 @@
"""detectScenes: PySceneDetect (adaptive) for shots, histogram grouping for scenes."""
from __future__ import annotations
import threading
from typing import Any
import cv2
import numpy as np
from scenedetect import AdaptiveDetector, SceneManager, StatsManager, open_video
from .. import media_io
from ..errors import Cancelled
from ..schemas import PathRequest
from .common import Ctx, clamp01, open_input
# Neighbouring shots whose middle frames correlate at least this much share a scene.
SCENE_HIST_CORRELATION = 0.6
ADAPTIVE_THRESHOLD = 3.0
class SceneDetector:
def detect(self, path: str, cancel: threading.Event) -> list[tuple[float, float, float]]:
"""Return shots as (start_s, end_s, score in [0,1])."""
video = open_video(path, backend="opencv")
stats = StatsManager()
manager = SceneManager(stats_manager=stats)
manager.add_detector(AdaptiveDetector(adaptive_threshold=ADAPTIVE_THRESHOLD))
stop = threading.Event()
def watch() -> None:
while not stop.wait(0.2):
if cancel.is_set():
manager.stop()
return
watcher = threading.Thread(target=watch, daemon=True)
watcher.start()
try:
manager.detect_scenes(video=video)
finally:
stop.set()
if cancel.is_set():
raise Cancelled()
scenes = manager.get_scene_list(start_in_scene=True)
ratio_key = next(
(k for k in AdaptiveDetector().get_metrics() if k.startswith("adaptive_ratio")), None
)
shots: list[tuple[float, float, float]] = []
for index, (start, end) in enumerate(scenes):
score = 1.0
if index > 0 and ratio_key:
value = (
stats.get_metrics(start.frame_num, [ratio_key])[0]
if stats.metrics_exist(start.frame_num, [ratio_key])
else None
)
score = clamp01((value or ADAPTIVE_THRESHOLD) / (2 * ADAPTIVE_THRESHOLD))
shots.append((start.seconds, end.seconds, score))
return shots
def load_scene_detector(_files: dict[str, str], _settings: Any) -> SceneDetector:
return SceneDetector()
def _histogram(frame: np.ndarray) -> np.ndarray:
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
hist = cv2.calcHist([hsv], [0, 1], None, [32, 16], [0, 180, 0, 256])
cv2.normalize(hist, hist)
return hist
def run(ctx: Ctx, body: PathRequest, cancel: threading.Event) -> dict[str, Any]:
real, info = open_input(ctx, body.path, cancel)
detector = ctx.registry.runtime("detectScenes")
raw = detector.detect(real, cancel)
duration_ms = int(round(info.duration_s * 1000))
if not raw:
raw = [(0.0, info.duration_s, 1.0)]
shots: list[dict[str, Any]] = []
for start_s, end_s, score in raw:
start_ms = int(round(start_s * 1000))
end_ms = max(int(round(end_s * 1000)), start_ms)
if duration_ms:
end_ms = min(end_ms, duration_ms)
shots.append({"startMs": start_ms, "endMs": end_ms, "score": float(score)})
scenes: list[dict[str, int]] = []
previous: np.ndarray | None = None
for shot in shots:
middle = (shot["startMs"] + shot["endMs"]) // 2
hist = _histogram(media_io.frame_at(real, info, middle, cancel, max_side=320))
same = (
previous is not None
and cv2.compareHist(previous, hist, cv2.HISTCMP_CORREL) >= SCENE_HIST_CORRELATION
)
if same and scenes:
scenes[-1]["endMs"] = shot["endMs"]
else:
scenes.append({"startMs": shot["startMs"], "endMs": shot["endMs"]})
previous = hist
return {"shots": shots, "scenes": scenes}

Some files were not shown because too many files have changed in this diff Show More