Compare commits

..

61 Commits

Author SHA1 Message Date
Claude
bab398cb52
wip(media): editor e2e and player fixes (unverified)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:29:59 +00:00
Claude
844f7fe7c1
feat(media): web project, editor, review, and brand kit views
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:19:45 +00:00
Claude
884d8903ca
feat(media): web view skeleton for Media Studio
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:04:47 +00:00
Claude
b4083844ce
feat(media): wave 2 integration
Composition root shared by API and worker, media job handlers on every job
host, HTTP byte routes, consent re-check before bundling, and auto-fix limited
to changes that keep protection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 19:04:01 +00:00
Claude
90f18cea52
merge: W2 privacy 2026-10-03 18:57:49 +00:00
Claude
d88e0de1bc
wip: media composition root 2026-10-03 18:57:49 +00:00
Claude
8d07ab17d4
feat(media): anonymization, marks, consent, and the render gate
prepareSources applies every face, mark, and region protection before the
compiler runs and fails closed when one cannot be honored: all face modes,
tracking with interpolation and grown holds over lost detections, reframe_out
crops with blur fallback, synthetic identity only through a licensed provider
(blur fallback otherwise), inpainting with a quality floor that falls back to
a cover, and person-bound marks re-found next to the face. The gate runs on
the render output: classifier at the profile rate plus cuts, face tracking
matched to people locally, pixel verification of every recorded protection,
consent and age proof (always for paid video), text and technical checks.
Auto-fix proposes only compliance repairs and stops after three rounds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:57:04 +00:00
Claude
2973f5074b
merge: W2 ingest
# Conflicts:
#	apps/desktop/src/preload.test.ts
2026-10-03 18:51:13 +00:00
Claude
21edc4add7
feat(media): wire marketing into the API, add segment results and teaser reset
Brand kits are validated through saveBrandKit and return their warnings;
manual metrics refit the teaser model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:50:45 +00:00
Claude
b416ff0fdf
feat(media): ingest job handlers with bounded concurrency and disk backpressure
media.sync, media.proxy, media.analyze, and media.cluster for Media Studio:
incremental metadata sync with deletion detection, 720p CFR proxies with
1 s keyframes, tone mapping, posters and sprites, sidecar analyses stored per
model version with scored segments, and sessions, themes, and person
suggestions. Proxy and analysis work is pumped in a small per-space window,
jobs step aside when their GPU or CPU permit is taken, and a full disk or an
unreachable sidecar pauses work instead of failing assets. Re-clustering
keeps collections the user merged or split. Fixes photo posters, which
never rendered because ffmpeg yields no frame when seeking a single image.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:50:30 +00:00
Claude
f29e6379cb
merge: W2 worktree-agent-a78be97f0347925a3 2026-10-03 18:47:30 +00:00
Claude
0e82982452
merge: W2 media-w2-render 2026-10-03 18:47:29 +00:00
Claude
227703a78a
merge: W2 worktree-agent-a346544dc365ebb55 2026-10-03 18:47:29 +00:00
Claude
4876c58fca
style(media): format zip test 2026-10-03 18:46:41 +00:00
Claude
7c744b98a0
feat(media): per-segment results and copy checks on every field 2026-10-03 18:46:36 +00:00
Claude
3ebe0fe47b
feat(media): ffmpeg renderer, render job and real render integration test
Fail closed when anonymization cannot prepare every clip, retry a failing
hardware encoder in software, verify each ffmpeg output exists, and fail
renders whose video or poster still carries source metadata. Cover parsing,
encoder detection, two-pass loudness, abort and cleanup, the render job
(auto-fix, cancel, outputs) on memory and Postgres, and a synthetic
16:9 to 9:16 render plus a region blur check with ffprobe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:44:58 +00:00
Claude
03e4961c9f
test(media): cover marketing plans, brand kits, bundles, publishing and learning 2026-10-03 18:43:37 +00:00
Claude
ffcc47710d
test(media): type the media router test client 2026-10-03 18:39:28 +00:00
Claude
e601b26292
feat(media): media API procedures, chunked uploads, and authorization tests 2026-10-03 18:37:51 +00:00
Claude
872fdd2194
feat(media): marketing service, brand kits, intro variants and bundle publisher 2026-10-03 18:36:53 +00:00
Claude
87d5be4daf
fix(desktop): iterate Photos persistent changes as a sequence
PHPersistentChangeFetchResult has no enumerateChanges; it is a Swift sequence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:31:45 +00:00
Claude
44396969fc
merge: main into media studio
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NLSCTnhge4umMnemGXWr1w
2026-10-03 18:30:25 +00:00
Claude
393137be9a
Merge remote-tracking branch 'origin/media-wip/w2-render' into media-w2-render 2026-10-03 18:29:04 +00:00
Claude
4721f00546
Merge remote-tracking branch 'origin/media-wip/w2-api' into worktree-agent-a346544dc365ebb55 2026-10-03 18:28:41 +00:00
Claude
d6957c4670
Merge remote-tracking branch 'origin/media-wip/w2-privacy' into worktree-agent-ac408b7c790dd9920 2026-10-03 18:28:38 +00:00
Claude
439de0d936
merge: media base b4222ac 2026-10-03 18:28:20 +00:00
Claude
3c13961948
Merge remote-tracking branch 'origin/media-wip/w2-marketing' into worktree-agent-a78be97f0347925a3 2026-10-03 18:28:15 +00:00
Claude
f4bf545c45
Merge remote-tracking branch 'origin/media-wip/w2-ingest' into worktree-agent-afca57be83bd9f7ba 2026-10-03 18:28:03 +00:00
Claude
7f6fc678c3
wip(media): w2-privacy partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:55 +00:00
Claude
3f9a601dba
wip(media): w2-marketing partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:53 +00:00
Claude
8f38c22119
wip(media): w2-render partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:51 +00:00
Claude
239a1eceb0
wip(media): w2-api partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:49 +00:00
Claude
55e129cb7e
wip(media): w2-ingest partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:47 +00:00
Claude
b4222ac5ec
feat(media): compose profile and CI job for the media sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:21:16 +00:00
Claude
bd5f637501
merge: W1-sidecar local media inference sidecar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:20:41 +00:00
Claude
5e2094f2ad
feat(media): local inference sidecar for Media Studio
FastAPI service in infra/media-worker implementing every
MediaInferenceProvider operation behind HTTP API v1, plus capabilities.
Bearer auth, MEDIA_ROOT realpath enforcement, strict request schemas,
cancellation on disconnect, GPU/CPU gates with 503 overflow, input
size/duration limits, lazy SHA256-pinned model downloads, fail-closed
redaction with atomic output, offline pytest suite with fake backends,
CPU/CUDA/ROCm Dockerfile and a license review of every model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:19:47 +00:00
Claude
a40582165a
feat(media): wire Apple Photos sync into the desktop app
Move the Photos bridge types into the desktop contract, expose
rakazoDesktop.media.photos from the preload, connect the sync runner in the
main process through an authenticated oRPC client, package and sign the
helper, and build and test it on macOS CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:15:53 +00:00
Claude
2556ed197e
feat(media): service contracts between media slices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:11:04 +00:00
Claude
475710e27e
feat(media): shared service deps and consent-gated model client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:10:02 +00:00
Claude
d43e374534
merge: W1-domain media domain logic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:39 +00:00
Claude
a6598f09d9
test(media): cover clustering, EDL, compiler, teaser, gate and marketing
Includes compiler snapshots, injection-proof escaping checks, and an
ffmpeg integration test on synthetic testsrc/sine inputs that is skipped
when ffmpeg is not installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
26d95a374e
feat(media): implement pure domain logic for Media Studio
Session and theme clustering, EDL validation and timeline mapping, the
EDL-to-ffmpeg compiler with caption files and filtergraph escaping,
teaser selection and learning, the compliance gate with compliance-only
auto-fix, and the marketing helpers (UTM, hashtags, CSV metrics, intro
proposals, brand-kit checks, upload checklist).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
1aab5ab3a8
feat(media): expose asset externalId, filter by ingest state, push device deletions
The desktop Photos sync needs to map awaiting assets back to Photos items,
query only awaiting uploads, and report deletions without a full relisting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:00:02 +00:00
Claude
51445e9442
merge: W1-desktop Apple Photos helper and sync
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:20 +00:00
Claude
0ee6c664d2
build(db): depend on @rakazo/media for coordinate rounding
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:02 +00:00
Claude
c877343218
merge: W1-db media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:32 +00:00
Claude
d3c6f262ad
feat(media): Apple Photos helper and client-pushed desktop sync
Adds the macOS photos-helper Swift CLI (PhotoKit listing with stable
cursors, persistent change tokens, iCloud-aware original export with a
path guard and SHA-256) plus XCTests for its PhotoKit-free core, and the
Electron main-process side: a validated helper client, a resumable
metadata push and chunked original upload against the media contract,
and IPC handlers that stay unavailable off macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:14 +00:00
Claude
5e155a0850
test(media): Postgres and unit tests for the media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c1c39b6e46
feat(media): space-scoped Media Studio repositories
Sources, assets with keyset search, analyses, segments, uploads,
collections with re-clustering, people with consent and complete
biometric removal, marks, brand kits, projects with versioned plans,
renders with atomic claims, publications, metrics and the funnel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c3f06e4a12
feat(media): wire media adapters and ffmpeg for CI media tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:53:43 +00:00
Claude
0c4e0202f3
merge: W1-adapters emulator, sidecar client, folder source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:59 +00:00
Claude
270ea8068a
feat(media): inference emulator, sidecar client, folder source and factory
Add the Media Studio adapters behind the frozen provider contracts: a
deterministic offline inference emulator driven by file-name markers that
renders real redactions with ffmpeg, a synthetic emulator library, the
bearer-auth HTTP client for the local media sidecar with validated
responses, typed errors, busy retries and timeouts, a lazily paginated
folder source that never follows symlinks, and the configuration factory.
A shared conformance suite runs against the emulator and, in the Docker
integration job, against the sidecar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:17 +00:00
Claude
f98fd9703c
feat(media): export motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:48:29 +00:00
Claude
759862fef4
merge: W1-motion motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:57 +00:00
Claude
b5cef21f44
feat(media): resvg motion graphics renderer with brand-kit templates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:24 +00:00
Claude
7ccc96ac11
feat(media): shared geo, path-safety and profile loading helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:34:04 +00:00
Claude
a663764ee8
feat(media): contracts and foundation for Media Studio
Freeze the shared contracts every Media Studio slice builds on: zod DTOs,
EDL, platform profile, brand kit, gate and render plan schemas; the media
RPC surface with a scaffold router; provider interfaces for sources,
inference, rendering, motion graphics and publishing; media background
jobs; the Prisma models and migration; and the @rakazo/media domain
package signatures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:32:52 +00:00
cynarAI
49431def83
feat: show the bot's live cursor and make the in-app browser work (#113)
Bots now announce every click and typing step before it runs, and the
web app draws a cursor in the bot's colour that glides to that point
over the live computer screen, ripples on clicks and shows typing.

- contracts: ComputerPointerPayload on the existing thread.computer event
- executor: computer_act and browser_act announce pointer steps
  (best effort); the page-browser helper reports element screen boxes,
  which are stripped before results reach the model
- web: agentPointerFeed + AgentCursor over the side-panel preview and
  the full-window screen, mapped through the noVNC letterbox
- in-app browser: native WebContentsView on desktop (own session, http(s)
  only, clamped bounds, back/forward, reload, DevTools, load errors) and
  a sandboxed iframe on the web; replaces the dead panel, the fake
  50%/40% cursors and the unimplemented devServer bridge


Claude-Session: https://claude.ai/code/session_01WiGooXtsKrZQmdszepRrNi

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-03 19:08:48 +02:00
cynarAI
3c3961f242
fix(api): keep host folders to the deployment owner (#112)
spaces.create linked any workspacePath to the new project, and
computer.files and computer.readFile then listed and read files under
that folder for anyone in the space. On a deployment with signups, any
member could link / or the API's own checkout and read whatever the API
process can read, such as its .env file. Linking also ran git in the
chosen folder and stored its remote URL.

A linked folder lives on the host, and the host is the deployment
owner's machine, the rule the onboarding scan already follows:

- spaces.create refuses a folder from anyone but the deployment owner,
  before the space is created or the folder is touched.
- Project files are listed and read only for the deployment owner, so a
  space a member linked before this change stays closed too. Space
  navigation no longer shows a linked folder's path and git remote to
  other members.
- A linked folder is stored as its real path, and listing and reading
  follow symlinks only while they stay inside it. Symlinks that leave
  the folder or lead nowhere are left out of a listing instead of
  failing it.
- The web New project dialog offers the folder field only to the
  deployment owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EyQw3AEaHYXegvgvTPAsi1
2026-10-03 18:32:48 +02:00
cynarAI
2b967df569
fix(api): turn repositories found by the onboarding scan into real projects (#116)
The scan's project phase called prisma.space with fields the Space model
does not have (title, path). Both calls threw, a per-repository
try/catch swallowed the errors, and no project was ever created. With
the field names fixed, the space would still have had no organization
or membership, so nobody could have seen it.

The phase now goes through createSpaceForMember and
wireProjectWorkspace, the helpers behind spaces.create: each repository
becomes a project of the deployment owner in their current
organization, linked to its folder with its git remote, branch and
head. It still runs only in the owner's scan with ONBOARDING_SCAN_HOME
set. A repository whose real folder one of the owner's projects already
links is skipped, so scanning again creates nothing; the phase stops at
the per-member space limit; and a second start while a scan runs (a
StrictMode remount, a retry) joins it instead of linking everything
twice. Discovery no longer descends into dot folders, where shell and
editor plugin checkouts would otherwise become projects. The Prisma
client is typed, so a field the schema lacks is a compile error again
instead of a swallowed one.

readGitCheckout loaded child_process with require(), which does not
exist in the ES module runtime the API runs in, so linking any folder
with a .git directory threw "require is not defined", in the scan and
in the New project dialog alike. It now imports spawn statically; the
module is only reachable through its server-only subpath, so browser
bundles still never load it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ebcq9seT9eCoHtqxuWEUzg
2026-10-03 18:22:10 +02:00
cynarAI
f8eabee997
fix(web): keep secondary shell text readable in the light theme (#117)
Secondary shell text was muted text dimmed further with opacity: the
section labels, the projects caption, the sidebar "+" icons, the privacy
panel, the trading subtitle and the Heute brief's Mission Control link.
Muted text measures 4.4:1 on the light sidebar, so these fell to 1.9 to
3.4:1 in the light theme; a few were below 4.5:1 in dark as well.

- Secondary sidebar text and icons share `rk-sidebar-muted`, the sidebar
  foreground at 65% that row times already used (5.0:1 on the light
  sidebar). Row times now use the class too.
- Section labels and space or project group titles use the dimmed
  foreground in light and keep the 80% muted color in dark; group titles
  move up from 60%, which was 3.4:1 in dark.
- The privacy panel heading is plain text with a success-colored check
  instead of emerald text, and its details pass 4.5:1 in both themes.
- Needs-you chips keep the warning tint and border; their label uses the
  foreground color in light.
- The Mission Control link uses the full muted color.
- E2E: the shell test checks every text in the sidebar except the logo,
  and the Heute brief, in both themes, and captures
  `shell-contrast-{light,dark}-heute`.
- AGENTS.md: agents make routine calls themselves, verify the result and
  report it, and ask only before irreversible actions or for product
  decisions without a sensible default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSVQGR4WU92LaddTP19GcQ
2026-10-03 17:56:44 +02:00
295 changed files with 58031 additions and 404 deletions

View File

@ -236,3 +236,15 @@ EVOO_COCKPIT_CORS_ORIGIN=
# Expo: English or Simplified Chinese (en | zh-CN). Overridden by SecureStore key
# rakazo.uiLocale from Account → Language.
# EXPO_PUBLIC_DEFAULT_UI_LOCALE=en
# Optional Media Studio worker (local ffmpeg + ML sidecar, compose profile "media").
# Without it, Media Studio stays hidden and media jobs are skipped.
# Compose: http://media-worker:8790. Bare-metal dev: http://127.0.0.1:8790.
# MEDIA_WORKER_URL=http://media-worker:8790
# At least 32 random characters.
# MEDIA_WORKER_TOKEN=REPLACE_WITH_RANDOM_TOKEN
# Private platform profiles that override the shipped ones. Keep outside the repo.
# MEDIA_PROFILES_DIR=~/.evoo/media-profiles
# Media jobs per worker process: model inference (GPU) and ffmpeg work (CPU).
# MEDIA_GPU_CONCURRENCY=1
# MEDIA_CPU_CONCURRENCY=2

View File

@ -170,6 +170,15 @@ jobs:
- run: pnpm db:generate
env:
DATABASE_URL: postgres://rakazo:rakazo@127.0.0.1:5433/rakazo
- name: Ensure ffmpeg for media tests
shell: bash
run: |
if command -v ffmpeg >/dev/null && command -v ffprobe >/dev/null; then exit 0; fi
if sudo -n true 2>/dev/null; then
sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
else
echo "::warning::ffmpeg is missing on this runner; media render tests will be skipped"
fi
- run: pnpm test
test-integration:

View File

@ -0,0 +1,32 @@
name: desktop-photos-helper
on:
pull_request:
paths: &photos-helper-paths
- ".github/workflows/desktop-photos-helper.yml"
- "apps/desktop/native/photos-helper/**"
push:
branches: [main]
paths: *photos-helper-paths
permissions:
contents: read
concurrency:
group: desktop-photos-helper-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
swift:
name: Photos helper build and XCTest
runs-on: [self-hosted, evoo-ci, macOS]
timeout-minutes: 20
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Build and test
working-directory: apps/desktop/native/photos-helper
run: |
set -euo pipefail
swift build -c release
swift test

32
.github/workflows/media-worker.yml vendored Normal file
View File

@ -0,0 +1,32 @@
name: media-worker
on:
pull_request:
paths: &media-worker-paths
- ".github/workflows/media-worker.yml"
- "infra/media-worker/**"
push:
branches: [main]
paths: *media-worker-paths
permissions:
contents: read
concurrency:
group: media-worker-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
sidecar:
name: Media sidecar
runs-on: [self-hosted, evoo-ci, Linux]
timeout-minutes: 30
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
- name: Build test image (CPU)
run: docker build --build-arg ACCELERATOR=cpu --target test -t rakazo/media-worker:ci-test infra/media-worker
- name: Pytest (offline, fake model backends)
run: docker run --rm --network none rakazo/media-worker:ci-test
- name: Build runtime image (CPU)
run: docker build --build-arg ACCELERATOR=cpu -t rakazo/media-worker:ci infra/media-worker

View File

@ -145,6 +145,14 @@ jobs:
umask 077
printf '%s\n' "$APPLE_API_KEY_P8" > "$APPLE_API_KEY"
chmod 600 "$APPLE_API_KEY"
- name: Build and test the universal Photos helper
if: runner.os == 'macOS'
working-directory: apps/desktop/native/photos-helper
run: |
set -euo pipefail
swift test
swift build -c release --arch arm64 --arch x86_64
lipo -verify_arch .build/apple/Products/Release/photos-helper arm64 x86_64
- name: Package signed and notarized universal macOS app
if: runner.os == 'macOS'
env:
@ -174,6 +182,10 @@ jobs:
grep -Fqx "provider: github" "$app/Contents/Resources/app-update.yml"
grep -Fqx "owner: elie222" "$app/Contents/Resources/app-update.yml"
grep -Fqx "repo: rakazo" "$app/Contents/Resources/app-update.yml"
helper="$app/Contents/Resources/bin/photos-helper"
codesign --verify --strict --verbose=2 "$helper"
codesign -d --entitlements - --xml "$helper" | grep -Fq "com.apple.security.personal-information.photos-library"
/usr/libexec/PlistBuddy -c 'Print :NSPhotoLibraryUsageDescription' "$app/Contents/Info.plist"
- name: Require Windows signing credentials
if: runner.os == 'Windows'

View File

@ -19,6 +19,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/db": "workspace:*",
"@rakazo/logging": "workspace:*",
"@rakazo/media": "workspace:*",
"@rakazo/memory": "workspace:*",
"hono": "4.13.7"
},

View File

@ -25,6 +25,8 @@ import {
createCloudAgentConnection,
createConnectorStack,
createJobReconciler,
createMediaJobHandlers,
createMediaServices,
createMessagingContextLoader,
createMessagingTeamChatSender,
createProfileConsolidationScheduler,
@ -92,6 +94,7 @@ import { cors } from "hono/cors";
import type { AppEnv } from "./env.js";
import { loadEnv } from "./env.js";
import { mountLocalSettings } from "./local-settings.js";
import { mountMediaHttpRoutes } from "./media-http.js";
import {
createMessagingInboundHandler,
teamChatSenderCanWakeMessageRoutines,
@ -414,6 +417,18 @@ export async function createApp(
shutdownSignal: shutdown.signal,
});
const mediaServices = createMediaServices({
config: env.media,
prisma,
jobs,
artifacts,
runtime,
realtime,
resolveModel: (scope) => executor.resolveModel(scope),
});
const mediaJobs = mediaServices
? createMediaJobHandlers(mediaServices, { signal: shutdown.signal })
: null;
const jobHandlers = createBackgroundJobHandlers({
executor,
prisma,
@ -421,6 +436,7 @@ export async function createApp(
home,
jobs,
events,
...(mediaJobs ? { media: mediaJobs.handlers } : {}),
workerId: "api",
runtime,
secretStore: secrets,
@ -438,6 +454,11 @@ export async function createApp(
});
await graphileWorkerHost.start(jobHandlers);
}
if (mediaJobs) {
void mediaJobs
.resume()
.catch((error) => getLogger().error("media ingest resume failed", error));
}
const reconciler = inMemoryJobs
? createJobReconciler({
prisma,
@ -469,6 +490,7 @@ export async function createApp(
remoteConnectors,
artifacts,
dataDir: env.dataDir,
mediaServices,
messaging: {
enabled: Boolean(messaging),
providers: messaging?.platforms().map((platform) => platform.provider) ?? [],
@ -583,6 +605,13 @@ export async function createApp(
if (actor) enrichLogContext({ "user.id": actor.userId, "space.id": actor.spaceId });
return actor;
});
mountMediaHttpRoutes(app, { services: mediaServices }, async (c) => {
const session = await auth.api.getSession({ headers: sessionHeaders(c.req.raw) });
if (!session?.user) return null;
return requireMembership(prisma, session.user.id, c.req.header("x-rakazo-space-id")).catch(
() => null,
);
});
mountWebhookHttpRoutes(app, { prisma, secrets, events, jobs });
// Shared with stop so a shutdown during retry delays does not restart polling.
let messagingStopped = false;

View File

@ -1,4 +1,6 @@
import type { MediaConfig } from "@rakazo/adapters";
import {
mediaConfigFromEnv,
resolveCloudAgentProvider,
resolveDeploymentModel,
resolveSandboxProvider,
@ -15,6 +17,8 @@ export { resolveCloudAgentProvider, resolveSandboxProvider } from "@rakazo/adapt
export interface AppEnv {
nodeEnv: string;
/** Media Studio; disabled unless a sidecar (or the scripted emulator) is configured. */
media: MediaConfig;
desktopStackToken?: string;
databaseUrl: string;
realtimeDatabaseUrl: string;
@ -106,6 +110,7 @@ export function loadEnv(source: NodeJS.ProcessEnv = process.env): AppEnv {
const updaterToken = optional(source.RAKAZO_UPDATER_TOKEN);
return {
nodeEnv: source.NODE_ENV ?? "",
media: mediaConfigFromEnv(source),
databaseUrl: required(source, "DATABASE_URL"),
realtimeDatabaseUrl: source.REALTIME_DATABASE_URL ?? required(source, "DATABASE_URL"),
desktopStackToken: optional(source.RAKAZO_DESKTOP_STACK_TOKEN),

265
apps/api/src/media-http.ts Normal file
View File

@ -0,0 +1,265 @@
import { createReadStream } from "node:fs";
import { mkdir, stat } from "node:fs/promises";
import path from "node:path";
import { Readable } from "node:stream";
import type { MediaServices } from "@rakazo/adapters";
import type { Actor } from "@rakazo/contracts";
import { IsolationError, MediaStateConflictError } from "@rakazo/db";
import { assertInside, PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import type { Context, Hono } from "hono";
import { cancelBody } from "./http-body.js";
import { MEDIA_CHUNK_TIMEOUT_MS, UploadChunkError, writeUploadChunk } from "./media-uploads.js";
/*
* Byte routes Media Studio cannot express as JSON RPC: chunk uploads and
* streaming of proxies, thumbnails, renders, and bundles. Every id is checked
* against the actor's space before a path is derived, and every path is
* resolved inside the media root.
*/
const ID = /^[A-Za-z0-9_-]{1,64}$/;
const FILE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/;
const CHUNK_INDEX = /^[0-9]{1,7}$/;
const CONTENT_TYPES: Record<string, string> = {
".mp4": "video/mp4",
".m4v": "video/mp4",
".mov": "video/quicktime",
".webm": "video/webm",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".png": "image/png",
".webp": "image/webp",
".gif": "image/gif",
".srt": "application/x-subrip",
".vtt": "text/vtt; charset=utf-8",
".json": "application/json",
".md": "text/markdown; charset=utf-8",
".txt": "text/plain; charset=utf-8",
".zip": "application/zip",
".m4a": "audio/mp4",
".wav": "audio/wav",
};
export function mediaContentType(fileName: string): string {
return CONTENT_TYPES[path.extname(fileName).toLowerCase()] ?? "application/octet-stream";
}
export type ByteRange = { start: number; end: number };
/**
* Parse a single `bytes=` range. Returns null for no (or an ignorable
* multi-part) range and "unsatisfiable" when it cannot be served.
*/
export function parseByteRange(
header: string | undefined,
size: number,
): ByteRange | null | "unsatisfiable" {
if (!header) return null;
const match = /^bytes=(\d*)-(\d*)$/.exec(header.trim());
if (!match) return header.includes(",") ? null : "unsatisfiable";
const [, first = "", last = ""] = match;
if (first === "" && last === "") return "unsatisfiable";
if (first === "") {
const suffix = Number(last);
if (suffix === 0 || size === 0) return "unsatisfiable";
return { start: Math.max(0, size - suffix), end: size - 1 };
}
const start = Number(first);
const end = last === "" ? size - 1 : Math.min(Number(last), size - 1);
if (start >= size || end < start) return "unsatisfiable";
return { start, end };
}
async function sendFile(
c: Context,
absolutePath: string,
options: { fileName: string; download?: boolean },
): Promise<Response> {
const info = await stat(absolutePath).catch(() => null);
if (!info?.isFile()) return c.json({ error: "Not found" }, 404);
const size = info.size;
const headers: Record<string, string> = {
"content-type": mediaContentType(options.fileName),
"accept-ranges": "bytes",
"cache-control": "private, max-age=300",
};
if (options.download) {
headers["content-disposition"] = `attachment; filename="${options.fileName}"`;
}
const range = parseByteRange(c.req.header("range"), size);
if (range === "unsatisfiable") {
return new Response(null, {
status: 416,
headers: { ...headers, "content-range": `bytes */${size}` },
});
}
if (c.req.method === "HEAD") {
return new Response(null, {
status: 200,
headers: { ...headers, "content-length": String(size) },
});
}
const start = range?.start ?? 0;
const end = range?.end ?? size - 1;
const body =
size === 0
? null
: (Readable.toWeb(
createReadStream(absolutePath, { start, end }),
) as ReadableStream<Uint8Array>);
return new Response(body, {
status: range ? 206 : 200,
headers: {
...headers,
"content-length": String(size === 0 ? 0 : end - start + 1),
...(range ? { "content-range": `bytes ${start}-${end}/${size}` } : {}),
},
});
}
function mediaHttpError(c: Context, error: unknown): Response {
if (error instanceof IsolationError || error instanceof PathOutsideRootError) {
return c.json({ error: "Not found" }, 404);
}
if (error instanceof UploadChunkError) return c.json({ error: error.message }, error.status);
if (error instanceof MediaStateConflictError) return c.json({ error: error.message }, 409);
if ((error as NodeJS.ErrnoException | null)?.code === "ENOENT") {
return c.json({ error: "Not found" }, 404);
}
throw error;
}
export function mountMediaHttpRoutes(
app: Hono,
deps: {
/** Null when Media Studio is not configured: every route answers 404. */
services: MediaServices | null;
chunkTimeoutMs?: number;
},
resolveActor: (c: Context) => Promise<Actor | null>,
) {
async function authorize(
c: Context,
): Promise<
{ actor: Actor; services: MediaServices; response?: undefined } | { response: Response }
> {
const actor = await resolveActor(c);
if (!actor) return { response: c.json({ error: "Unauthorized" }, 401) };
if (!deps.services)
return { response: c.json({ error: "Media Studio is not configured" }, 404) };
return { actor, services: deps.services };
}
async function mediaFile(services: MediaServices, relative: string | null) {
if (!relative) return null;
await mkdir(services.deps.mediaRoot, { recursive: true });
return resolveInside(services.deps.mediaRoot, relative);
}
app.put("/api/media/uploads/:uploadId/chunks/:index", async (c) => {
const auth = await authorize(c);
if (auth.response) {
cancelBody(c.req.raw.body);
return auth.response;
}
const uploadId = c.req.param("uploadId");
const index = c.req.param("index");
if (!ID.test(uploadId) || !CHUNK_INDEX.test(index)) {
cancelBody(c.req.raw.body);
return c.json({ error: "Not found" }, 404);
}
const lengthHeader = c.req.header("content-length");
const declared = lengthHeader && /^[0-9]+$/.test(lengthHeader) ? Number(lengthHeader) : null;
const timeout = AbortSignal.timeout(deps.chunkTimeoutMs ?? MEDIA_CHUNK_TIMEOUT_MS);
const body = c.req.raw.body;
try {
const upload = await writeUploadChunk({
services: auth.services,
spaceId: auth.actor.spaceId,
uploadId,
index: Number(index),
body,
declaredLength: declared,
signal: AbortSignal.any([timeout, c.req.raw.signal]),
});
return c.json(upload);
} catch (error) {
cancelBody(body);
if (timeout.aborted) return c.json({ error: "Chunk took too long" }, 408);
return mediaHttpError(c, error);
}
});
app.get("/api/media/renders/:renderId/files/:fileName", async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const renderId = c.req.param("renderId");
const fileName = c.req.param("fileName");
if (!ID.test(renderId) || !FILE_NAME.test(fileName) || fileName.includes("..")) {
return c.json({ error: "Not found" }, 404);
}
try {
const render = await auth.services.deps.repos.getRenderRecord(auth.actor.spaceId, renderId);
const download = c.req.query("download") === "1";
// Only a render that passed its gate leaves the app as a download.
if (download && render.status !== "done") return c.json({ error: "Not found" }, 404);
const absolute = await auth.services.render.outputPath(auth.actor, renderId, fileName);
if (!absolute) return c.json({ error: "Not found" }, 404);
await mkdir(auth.services.deps.mediaRoot, { recursive: true });
const inside = await assertInside(
await resolveInside(
auth.services.deps.mediaRoot,
`${auth.actor.spaceId}/renders/${renderId}`,
),
absolute,
);
return await sendFile(c, inside, { fileName, download });
} catch (error) {
return mediaHttpError(c, error);
}
});
for (const kind of ["thumb", "proxy"] as const) {
app.get(`/api/media/assets/:assetId/${kind}`, async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const assetId = c.req.param("assetId");
if (!ID.test(assetId)) return c.json({ error: "Not found" }, 404);
try {
const asset = await auth.services.deps.repos.getAssetRecord(auth.actor.spaceId, assetId);
const relative = kind === "thumb" ? asset.thumbPath : asset.proxyPath;
const absolute = await mediaFile(auth.services, relative);
if (!absolute) return c.json({ error: "Not found" }, 404);
return await sendFile(c, absolute, { fileName: path.basename(absolute) });
} catch (error) {
return mediaHttpError(c, error);
}
});
}
app.get("/api/media/bundles/:publicationId", async (c) => {
const auth = await authorize(c);
if (auth.response) return auth.response;
const publicationId = c.req.param("publicationId");
if (!ID.test(publicationId)) return c.json({ error: "Not found" }, 404);
try {
const publication = await auth.services.deps.repos.getPublicationRecord(
auth.actor.spaceId,
publicationId,
);
if (publication.bundleArtifactId) return c.json({ error: "Not found" }, 404);
const absolute = await mediaFile(
auth.services,
`${auth.actor.spaceId}/bundles/${publication.id}/bundle.zip`,
);
if (!absolute) return c.json({ error: "Not found" }, 404);
return await sendFile(c, absolute, {
fileName: `bundle-${publication.id}.zip`,
download: true,
});
} catch (error) {
return mediaHttpError(c, error);
}
});
}

View File

@ -0,0 +1,389 @@
import type { MediaInferenceProvider } from "@rakazo/adapter-kit";
import type {
MediaActor,
MediaIngestService,
MediaMarketingService,
MediaPrivacyService,
MediaRenderService,
MediaServiceDeps,
MediaServices,
} from "@rakazo/adapters";
import type { PlatformProfile } from "@rakazo/contracts";
import { mediaRpcShapes } from "@rakazo/contracts";
import type { MediaRepos, PrismaClient } from "@rakazo/db";
/*
* In-memory stand-ins for the media service slices, for API tests only. They
* record what the router hands them and do the minimum through the real
* repositories; nothing here renders, detects, or publishes.
*/
export interface MediaServiceCall {
service: "ingest" | "privacy" | "render" | "marketing";
method: string;
actor: MediaActor;
input: unknown;
}
export interface FakeMediaServicesOptions {
repos: MediaRepos;
mediaRoot: string;
prisma?: PrismaClient;
inference?: MediaInferenceProvider | null;
profiles?: PlatformProfile[];
now?: () => Date;
/** Render id → absolute output path the fake render service reports. */
renderOutputs?: Map<string, string>;
onUploadCompleted?: (
actor: MediaActor,
input: { assetId: string; tempPath: string; sha256: string; fileName: string },
) => Promise<void>;
}
export function fakeMediaServices(options: FakeMediaServicesOptions): {
services: MediaServices;
calls: MediaServiceCall[];
} {
const calls: MediaServiceCall[] = [];
const { repos } = options;
const record = (
service: MediaServiceCall["service"],
method: string,
actor: MediaActor,
input: unknown,
) => {
calls.push({ service, method, actor: { spaceId: actor.spaceId, userId: actor.userId }, input });
};
const ingest: MediaIngestService = {
async pushItems(actor, input) {
record("ingest", "pushItems", actor, input);
return { created: 0, updated: 0, unchanged: 0, deleted: 0 };
},
async requestSync(actor, sourceId) {
record("ingest", "requestSync", actor, sourceId);
},
async uploadCompleted(actor, input) {
record("ingest", "uploadCompleted", actor, input);
await options.onUploadCompleted?.(actor, input);
},
async recluster(actor) {
record("ingest", "recluster", actor, null);
},
async removeSource(actor, sourceId) {
record("ingest", "removeSource", actor, sourceId);
await repos.removeSource(actor.spaceId, sourceId);
},
};
const privacy: MediaPrivacyService = {
async prepareSources() {
throw new Error("prepareSources is not part of the API fakes");
},
async gate() {
throw new Error("gate is not part of the API fakes");
},
async createMark(actor, input) {
record("privacy", "createMark", actor, input);
return repos.createMark(actor.spaceId, input);
},
async updateMark(actor, input) {
record("privacy", "updateMark", actor, input);
const { markId, ...patch } = input;
return repos.updateMark(actor.spaceId, markId, patch);
},
async removePerson(actor, personId) {
record("privacy", "removePerson", actor, personId);
await repos.removePersonCompletely(actor.spaceId, personId);
},
};
const render: MediaRenderService = {
async start(actor, input) {
record("render", "start", actor, input);
return repos.createRender(actor.spaceId, { ...input, createdBy: actor.userId });
},
async cancel(actor, renderId) {
record("render", "cancel", actor, renderId);
return repos.cancelRender(actor.spaceId, renderId);
},
async outputPath(actor, renderId, fileName) {
record("render", "outputPath", actor, { renderId, fileName });
const file = options.renderOutputs?.get(renderId);
return file?.endsWith(`/${fileName}`) ? file : null;
},
};
const marketing: MediaMarketingService = {
async autoPlan(actor, input) {
record("marketing", "autoPlan", actor, input);
return [];
},
async submitPlan(actor, input) {
record("marketing", "submitPlan", actor, input);
return { ok: false, errors: [] };
},
async introVariants(actor, input) {
record("marketing", "introVariants", actor, input);
return [];
},
async thumbnails(actor, input) {
record("marketing", "thumbnails", actor, input);
return [];
},
async copy(actor, input) {
record("marketing", "copy", actor, input);
const render = await repos.getRender(actor.spaceId, input.renderId);
return {
profileId: render.profileId,
language: input.language,
title: "Title",
hook: "Hook",
caption: "Caption",
hashtags: [],
issues: [],
};
},
async createPublication(actor, input) {
record("marketing", "createPublication", actor, input);
return repos.createPublication(actor.spaceId, actor.userId, {
...input,
trackingUrl: null,
scheduledAt: input.scheduledAt ? new Date(input.scheduledAt) : null,
});
},
async bundle(actor, publicationId) {
record("marketing", "bundle", actor, publicationId);
return repos.markPublicationBundled(actor.spaceId, publicationId, null);
},
async cancelPublication(actor, publicationId) {
record("marketing", "cancelPublication", actor, publicationId);
return repos.cancelPublication(actor.spaceId, publicationId);
},
async importMetricsCsv(actor, input) {
record("marketing", "importMetricsCsv", actor, input);
return { imported: 0, skipped: [] };
},
};
const deps = {
prisma: options.prisma,
repos,
jobs: { publish: async () => undefined },
inference: options.inference ?? null,
renderer: {
id: "fake",
hardwareEncoders: async () => ["h264_fake"],
render: async () => {
throw new Error("render is not part of the API fakes");
},
},
motion: { id: "fake", templates: () => [], render: async () => ({ path: "", hasAlpha: true }) },
publisher: { id: "fake", publish: async () => ({ status: "bundled", externalUrl: null }) },
shortener: null,
sources: {},
mediaRoot: options.mediaRoot,
profiles: async () => ({
profiles: new Map(
(options.profiles ?? []).map((profile) => [
profile.id,
{ profile, origin: "builtin" as const },
]),
),
errors: [],
}),
artifacts: {},
model: { generate: async () => null },
publish: async () => undefined,
now: options.now ?? (() => new Date()),
} as unknown as MediaServiceDeps;
return { services: { deps, ingest, privacy, render, marketing }, calls };
}
/** Every RPC path of the media contract, e.g. "sources/list". */
export function mediaProcedurePaths(): string[] {
const paths: string[] = [];
const walk = (node: Record<string, unknown>, prefix: string[]) => {
for (const [key, value] of Object.entries(node)) {
const child = value as Record<string, unknown>;
if ("input" in child && "output" in child) paths.push([...prefix, key].join("/"));
else walk(child, [...prefix, key]);
}
};
walk(mediaRpcShapes, []);
return paths;
}
export interface MediaSampleIds {
sourceId: string;
assetId: string;
secondAssetId: string;
segmentId: string;
trackId: string;
personId: string;
markId: string;
collectionId: string;
brandKitId: string;
projectId: string;
planId: string;
renderId: string;
publicationId: string;
mappingId: string;
uploadId: string;
}
/**
* How a procedure must treat another space's ids: refuse them as not found,
* answer only from the caller's own space, or be covered by a dedicated test.
*/
export type ForeignExpectation = "not_found" | "scoped" | "dedicated";
/** A valid input for every media procedure, built from one space's ids. */
export function sampleMediaInputs(
ids: MediaSampleIds,
kit: unknown,
): Record<string, { input: unknown; foreign: ForeignExpectation }> {
const at = "2026-09-01T10:00:00.000Z";
return {
capabilities: { input: {}, foreign: "scoped" },
"sources/list": { input: {}, foreign: "scoped" },
"sources/createFolder": { input: { label: "Footage", path: "/" }, foreign: "dedicated" },
"sources/registerClient": {
input: { kind: "ios_library", label: "Phone", deviceId: "device-sample" },
foreign: "scoped",
},
"sources/pushItems": {
input: { sourceId: ids.sourceId, items: [], syncToken: null, complete: true },
foreign: "not_found",
},
"sources/sync": { input: { sourceId: ids.sourceId }, foreign: "not_found" },
"sources/remove": { input: { sourceId: ids.sourceId }, foreign: "not_found" },
"assets/search": { input: {}, foreign: "scoped" },
"assets/get": { input: { assetId: ids.assetId }, foreign: "not_found" },
"assets/setPaidOnly": {
input: { segmentIds: [ids.segmentId], paidOnly: true },
foreign: "not_found",
},
"assets/requestUpload": { input: { assetIds: [ids.assetId] }, foreign: "not_found" },
"uploads/begin": {
input: {
assetId: ids.secondAssetId,
sizeBytes: 10,
sha256: "a".repeat(64),
fileName: "clip.mp4",
},
foreign: "not_found",
},
"uploads/status": { input: { uploadId: ids.uploadId }, foreign: "not_found" },
"uploads/complete": { input: { uploadId: ids.uploadId }, foreign: "not_found" },
"collections/list": { input: {}, foreign: "scoped" },
"collections/get": { input: { collectionId: ids.collectionId }, foreign: "not_found" },
"collections/rename": {
input: { collectionId: ids.collectionId, title: "Renamed" },
foreign: "not_found",
},
"collections/merge": {
input: { collectionIds: [ids.collectionId, `${ids.collectionId}x`] },
foreign: "not_found",
},
"collections/split": {
input: { collectionId: ids.collectionId, atAssetId: ids.assetId },
foreign: "not_found",
},
"collections/createManual": {
input: { title: "Picks", assetIds: [ids.assetId] },
foreign: "not_found",
},
"collections/recluster": { input: {}, foreign: "scoped" },
"people/list": { input: {}, foreign: "scoped" },
"people/update": { input: { personId: ids.personId, displayName: "X" }, foreign: "not_found" },
"people/merge": {
input: { personIds: [ids.personId, `${ids.personId}x`] },
foreign: "not_found",
},
"people/assignTrack": { input: { trackId: ids.trackId, personId: null }, foreign: "not_found" },
"people/recordConsent": {
input: {
personId: ids.personId,
consentArtifactId: null,
consentRecordedAt: at,
ageVerificationMethod: "platform_verified",
ageVerifiedAt: at,
},
foreign: "not_found",
},
"people/remove": { input: { personId: ids.personId }, foreign: "not_found" },
"marks/list": { input: { assetId: ids.assetId }, foreign: "scoped" },
"marks/create": {
input: { kind: "tattoo", assetId: ids.assetId, prompt: { tMs: 0 }, treatment: "blur" },
foreign: "not_found",
},
"marks/update": { input: { markId: ids.markId, label: "x" }, foreign: "not_found" },
"marks/remove": { input: { markId: ids.markId }, foreign: "not_found" },
"brandKits/list": { input: {}, foreign: "scoped" },
"brandKits/upsert": { input: { id: ids.brandKitId, kit }, foreign: "not_found" },
"brandKits/remove": { input: { id: ids.brandKitId }, foreign: "not_found" },
"brandKits/templates": { input: {}, foreign: "scoped" },
"brandKits/introVariants": { input: { id: ids.brandKitId }, foreign: "not_found" },
"profiles/list": { input: {}, foreign: "scoped" },
"profiles/get": { input: { profileId: "example-short-vertical" }, foreign: "scoped" },
"projects/list": { input: {}, foreign: "scoped" },
"projects/get": { input: { projectId: ids.projectId }, foreign: "not_found" },
"projects/create": {
input: {
title: "Teaser",
goal: "teaser",
collectionIds: [ids.collectionId],
profileIds: ["example-short-vertical"],
},
foreign: "not_found",
},
"projects/update": { input: { projectId: ids.projectId, title: "X" }, foreign: "not_found" },
"projects/remove": { input: { projectId: ids.projectId }, foreign: "not_found" },
"projects/autoPlan": { input: { projectId: ids.projectId }, foreign: "not_found" },
"plans/submit": { input: { projectId: ids.projectId, edl: {} }, foreign: "not_found" },
"renders/start": { input: { planId: ids.planId, quality: "draft" }, foreign: "not_found" },
"renders/get": { input: { renderId: ids.renderId }, foreign: "not_found" },
"renders/cancel": { input: { renderId: ids.renderId }, foreign: "not_found" },
"renders/thumbnails": { input: { renderId: ids.renderId }, foreign: "not_found" },
"marketing/copy": { input: { renderId: ids.renderId }, foreign: "not_found" },
"publications/list": { input: { projectId: ids.projectId }, foreign: "scoped" },
"publications/create": { input: { renderId: ids.renderId }, foreign: "not_found" },
"publications/bundle": { input: { publicationId: ids.publicationId }, foreign: "not_found" },
"publications/markPublished": {
input: { publicationId: ids.publicationId, externalUrl: null },
foreign: "not_found",
},
"publications/cancel": { input: { publicationId: ids.publicationId }, foreign: "not_found" },
"metrics/mappings": { input: {}, foreign: "scoped" },
"metrics/saveMapping": {
input: {
id: ids.mappingId,
name: "Imported",
columns: { publication: "id", observedAt: "date" },
publicationKey: "id",
},
foreign: "not_found",
},
"metrics/importCsv": {
input: { mappingId: ids.mappingId, csv: "id,date" },
foreign: "not_found",
},
"metrics/record": {
input: {
publicationId: ids.publicationId,
observedAt: at,
views: 1,
clicks: 0,
conversions: 0,
revenueCents: 0,
currency: "EUR",
},
foreign: "not_found",
},
"metrics/funnel": { input: { projectId: ids.projectId }, foreign: "not_found" },
"metrics/segments": { input: { projectId: ids.projectId }, foreign: "not_found" },
"metrics/resetTeaserModel": { input: {}, foreign: "scoped" },
};
}

View File

@ -0,0 +1,244 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { mkdir, open, rename, rm, stat } from "node:fs/promises";
import path from "node:path";
import type { MediaActor, MediaServices } from "@rakazo/adapters";
import type { MediaUpload } from "@rakazo/contracts";
import type { MediaRepos } from "@rakazo/db";
import { MediaStateConflictError } from "@rakazo/db";
import { resolveInside } from "@rakazo/media/node";
/*
* Resumable chunked uploads. Each chunk lands in its own file
* (<spaceId>/uploads/<uploadId>/chunks/<index>.part, written to a temp name and
* renamed, so a repeated or concurrent PUT of the same index is harmless).
* Completion concatenates the chunks in order into <spaceId>/uploads/<uploadId>/file
* while hashing, verifies size and SHA-256, and hands the file to ingest.
*
* Limits: the contract caps the total size, every chunk must have its exact
* length, one chunk must arrive within MEDIA_CHUNK_TIMEOUT_MS, and an upload
* that saw no progress for MEDIA_UPLOAD_IDLE_MS expires and must start over.
*/
/** A single chunk must arrive within this window; slow links resume per chunk. */
export const MEDIA_CHUNK_TIMEOUT_MS = 10 * 60_000;
/** An open upload without a new chunk for this long is failed and its bytes removed. */
export const MEDIA_UPLOAD_IDLE_MS = 24 * 60 * 60_000;
export class UploadChunkError extends Error {
constructor(
readonly status: 400 | 409 | 413,
message: string,
) {
super(message);
this.name = "UploadChunkError";
}
}
type UploadRow = Awaited<ReturnType<MediaRepos["getUploadRecord"]>>;
type UploadSize = Pick<UploadRow, "sizeBytes" | "chunkBytes">;
export function uploadChunkCount(row: UploadSize): number {
return Number((row.sizeBytes + BigInt(row.chunkBytes) - 1n) / BigInt(row.chunkBytes));
}
/** Exact byte length chunk `index` must have: every chunk is full except the last. */
export function expectedChunkBytes(row: UploadSize, index: number): number {
const count = uploadChunkCount(row);
if (index < count - 1) return row.chunkBytes;
return Number(row.sizeBytes - BigInt(row.chunkBytes) * BigInt(count - 1));
}
function uploadDir(row: Pick<UploadRow, "spaceId" | "id">): string {
return `${row.spaceId}/uploads/${row.id}`;
}
async function mediaPath(mediaRoot: string, relative: string): Promise<string> {
await mkdir(mediaRoot, { recursive: true });
return resolveInside(mediaRoot, relative);
}
async function chunksDir(mediaRoot: string, row: UploadRow): Promise<string> {
const dir = await mediaPath(mediaRoot, `${uploadDir(row)}/chunks`);
await mkdir(dir, { recursive: true });
return dir;
}
async function removeUploadFiles(mediaRoot: string, row: UploadRow): Promise<void> {
await rm(await mediaPath(mediaRoot, uploadDir(row)), { recursive: true, force: true });
}
function isIdle(services: MediaServices, row: UploadRow): boolean {
return services.deps.now().getTime() - row.updatedAt.getTime() > MEDIA_UPLOAD_IDLE_MS;
}
/** Fail an idle open upload and drop its bytes. Returns true when it expired. */
async function expireIfIdle(services: MediaServices, row: UploadRow): Promise<boolean> {
if (row.status !== "open" || !isIdle(services, row)) return false;
await services.deps.repos.failUpload(row.spaceId, row.id).catch((error: unknown) => {
// A concurrent request already closed it; the files still go.
if (!(error instanceof MediaStateConflictError)) throw error;
});
await removeUploadFiles(services.deps.mediaRoot, row);
return true;
}
/**
* Open an upload, or resume the open one for the same asset and content. An
* idle one expires first so the client starts over with a fresh upload.
*/
export async function beginMediaUpload(
services: MediaServices,
actor: MediaActor,
input: { assetId: string; sizeBytes: number; sha256: string; fileName: string },
): Promise<MediaUpload> {
const { repos } = services.deps;
const asset = await repos.getAssetRecord(actor.spaceId, input.assetId);
// Device items wait for their original; files read from a source never upload.
if (asset.storage !== "remote" && asset.ingestState !== "awaiting_upload") {
throw new MediaStateConflictError("This item does not need an upload");
}
const begin = () =>
repos.beginUpload(actor.spaceId, { ...input, assetId: asset.id, userId: actor.userId });
const upload = await begin();
const row = await repos.getUploadRecord(actor.spaceId, upload.id);
return (await expireIfIdle(services, row)) ? begin() : upload;
}
/**
* Stream one chunk body to disk without buffering more than one network read.
* Rejects bodies longer or shorter than the chunk's exact size.
*/
export async function writeUploadChunk(input: {
services: MediaServices;
spaceId: string;
uploadId: string;
index: number;
body: ReadableStream<Uint8Array> | null;
declaredLength: number | null;
/** Aborting stops reading the body; the chunk is then discarded as incomplete. */
signal?: AbortSignal;
}): Promise<MediaUpload> {
const { services } = input;
const { repos, mediaRoot } = services.deps;
const row = await repos.getUploadRecord(input.spaceId, input.uploadId);
if (await expireIfIdle(services, row)) throw new UploadChunkError(409, "Upload expired");
if (row.status !== "open") throw new UploadChunkError(409, "Upload is not open");
const count = uploadChunkCount(row);
if (!Number.isInteger(input.index) || input.index < 0 || input.index >= count) {
throw new UploadChunkError(400, "Chunk index out of range");
}
const expected = expectedChunkBytes(row, input.index);
if (input.declaredLength !== null && input.declaredLength > expected) {
throw new UploadChunkError(413, "Chunk is too large");
}
if (!input.body) throw new UploadChunkError(400, "Chunk is empty");
const dir = await chunksDir(mediaRoot, row);
const finalPath = path.join(dir, `${input.index}.part`);
const tempPath = path.join(dir, `${input.index}.${randomUUID()}.tmp`);
const handle = await open(tempPath, "wx");
let written = 0;
let ok = false;
const reader = input.body.getReader();
const stop = () => void reader.cancel().catch(() => undefined);
input.signal?.addEventListener("abort", stop, { once: true });
try {
if (input.signal?.aborted) stop();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
written += value.byteLength;
if (written > expected) {
void reader.cancel().catch(() => undefined);
throw new UploadChunkError(413, "Chunk is too large");
}
await handle.write(value);
}
if (written !== expected) throw new UploadChunkError(400, "Chunk is incomplete");
ok = true;
} finally {
input.signal?.removeEventListener("abort", stop);
reader.releaseLock();
await handle.close();
if (!ok) await rm(tempPath, { force: true });
}
await rename(tempPath, finalPath);
return repos.markChunkReceived(input.spaceId, input.uploadId, input.index);
}
const completing = new Set<string>();
/**
* Assemble a fully received upload, verify it, hand it to ingest, and close it.
* A checksum mismatch fails the upload so the client starts a fresh one.
*/
export async function completeMediaUpload(
services: MediaServices,
actor: MediaActor,
uploadId: string,
): Promise<MediaUpload> {
const { repos, mediaRoot } = services.deps;
const row = await repos.getUploadRecord(actor.spaceId, uploadId);
if (row.status === "complete") return repos.uploadStatus(actor.spaceId, uploadId);
if (await expireIfIdle(services, row)) throw new UploadChunkError(409, "Upload expired");
if (row.status !== "open") throw new MediaStateConflictError("Upload is not open");
const received = new Set(row.receivedChunks);
const count = uploadChunkCount(row);
for (let index = 0; index < count; index++) {
if (!received.has(index)) throw new MediaStateConflictError("Upload is missing chunks");
}
if (completing.has(row.id)) throw new MediaStateConflictError("Upload is already completing");
completing.add(row.id);
try {
const dir = await chunksDir(mediaRoot, row);
const relativeFile = `${uploadDir(row)}/file`;
const filePath = await mediaPath(mediaRoot, relativeFile);
const tempFile = await mediaPath(mediaRoot, `${uploadDir(row)}/file.${randomUUID()}.tmp`);
const hash = createHash("sha256");
let total = 0;
const out = await open(tempFile, "wx");
try {
for (let index = 0; index < count; index++) {
const chunkPath = path.join(dir, `${index}.part`);
const info = await stat(chunkPath).catch(() => null);
if (!info || info.size !== expectedChunkBytes(row, index)) {
throw new MediaStateConflictError("Upload is missing chunks");
}
for await (const buffer of createReadStream(chunkPath) as AsyncIterable<Buffer>) {
hash.update(buffer);
total += buffer.byteLength;
await out.write(buffer);
}
}
} catch (error) {
await out.close();
await rm(tempFile, { force: true });
throw error;
}
await out.close();
const sha256 = hash.digest("hex");
if (sha256 !== row.sha256 || BigInt(total) !== row.sizeBytes) {
await removeUploadFiles(mediaRoot, row);
await repos.failUpload(actor.spaceId, uploadId);
throw new UploadChunkError(400, "Upload does not match its checksum");
}
await rename(tempFile, filePath);
await rm(dir, { recursive: true, force: true });
await services.ingest.uploadCompleted(actor, {
assetId: row.assetId,
tempPath: filePath,
sha256,
fileName: row.fileName,
});
// Ingest normally moves the file into the asset folder and closes the
// upload itself; otherwise the assembled file stays as the original.
const after = await repos.getUploadRecord(actor.spaceId, uploadId);
if (after.status === "complete") {
await removeUploadFiles(mediaRoot, row);
return repos.uploadStatus(actor.spaceId, uploadId);
}
return repos.completeUpload(actor.spaceId, uploadId, { storagePath: relativeFile });
} finally {
completing.delete(row.id);
}
}

View File

@ -0,0 +1,516 @@
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createRouterClient, ORPCError } from "@orpc/server";
import type { Actor } from "@rakazo/contracts";
import type { MediaRepos } from "@rakazo/db";
import { createMediaRepos } from "@rakazo/db";
import { Hono } from "hono";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import type { MediaTestDb, SeededMediaSpace } from "../../../packages/db/src/media-fixtures.js";
import {
openMediaTestDb,
seedMediaSpace,
sourceItem,
testBrandKit,
} from "../../../packages/db/src/media-fixtures.js";
import { testProfile } from "../../../packages/media/src/test-fixtures.js";
import { createMediaRouter } from "./media.js";
import { mountMediaHttpRoutes } from "./media-http.js";
import type { MediaServiceCall } from "./media-test-fakes.js";
import { fakeMediaServices, sampleMediaInputs } from "./media-test-fakes.js";
import { MEDIA_UPLOAD_IDLE_MS } from "./media-uploads.js";
const databaseUrl = process.env.DATABASE_URL;
const describePostgres =
process.env.VERIFY_DATABASE && databaseUrl ? describe.sequential : describe.skip;
type Space = { userId: string; spaceId: string };
function actorOf(space: Space, isDeploymentOwner = false): Actor {
return {
userId: space.userId,
spaceId: space.spaceId,
email: `${space.userId}@rakazo.test`,
isDeploymentOwner,
};
}
function procedure(client: unknown, rpcPath: string) {
let node = client;
for (const key of rpcPath.split("/")) node = (node as Record<string, unknown>)[key];
return node as (input: unknown) => Promise<unknown>;
}
async function rpcCode(promise: Promise<unknown>): Promise<string> {
try {
await promise;
} catch (error) {
if (error instanceof ORPCError) return error.code;
throw error;
}
return "OK";
}
function sha256(bytes: Uint8Array): string {
return createHash("sha256").update(bytes).digest("hex");
}
describePostgres("media API authorization and uploads (PostgreSQL)", () => {
let db: MediaTestDb;
let repos: MediaRepos;
let mediaRoot: string;
let a: Space;
let b: Space;
let seededA: SeededMediaSpace;
let seededB: SeededMediaSpace;
let calls: MediaServiceCall[];
let renderOutputs: Map<string, string>;
let rpc: (actor: Actor) => unknown;
let app: Hono;
let currentActor: Actor | null;
function setup(options: { now?: () => Date; chunkTimeoutMs?: number } = {}) {
const fake = fakeMediaServices({
repos,
mediaRoot,
prisma: db.prisma,
profiles: [testProfile()],
renderOutputs,
...(options.now ? { now: options.now } : {}),
});
calls = fake.calls;
const router = createMediaRouter({
prisma: db.prisma,
services: fake.services,
dataDir: path.join(mediaRoot, ".."),
});
rpc = (actor) => createRouterClient(router, { context: { actor } });
app = new Hono();
mountMediaHttpRoutes(
app,
{
services: fake.services,
...(options.chunkTimeoutMs ? { chunkTimeoutMs: options.chunkTimeoutMs } : {}),
},
async () => currentActor,
);
}
async function snapshot(spaceId: string) {
const p = db.prisma;
const where = { spaceId };
const orderBy = { id: "asc" as const };
return JSON.stringify(
[
await p.mediaSource.findMany({ where, orderBy }),
await p.mediaAsset.findMany({ where, orderBy }),
await p.mediaUpload.findMany({ where, orderBy }),
await p.mediaSegment.findMany({ where, orderBy }),
await p.mediaAnalysis.findMany({ where, orderBy }),
await p.mediaCollection.findMany({ where, orderBy }),
await p.mediaCollectionItem.findMany({
where: { collection: where },
orderBy: [{ collectionId: "asc" }, { assetId: "asc" }],
}),
await p.mediaPerson.findMany({ where, orderBy }),
await p.mediaFaceTrack.findMany({ where, orderBy }),
await p.mediaMark.findMany({ where, orderBy }),
await p.mediaBrandKit.findMany({ where, orderBy }),
await p.mediaProject.findMany({ where, orderBy }),
await p.mediaEditPlan.findMany({ where, orderBy }),
await p.mediaRender.findMany({ where, orderBy }),
await p.mediaPublication.findMany({ where, orderBy }),
await p.mediaMetric.findMany({ where, orderBy }),
await p.mediaMetricMapping.findMany({ where, orderBy }),
],
(_key, value) => (typeof value === "bigint" ? value.toString() : value),
);
}
async function artifactIn(space: Space): Promise<string> {
const artifact = await db.prisma.artifact.create({
data: {
spaceId: space.spaceId,
userId: space.userId,
name: "consent.pdf",
mimeType: "application/pdf",
size: 1,
hash: "0".repeat(64),
storageKey: `test/${space.spaceId}/consent.pdf`,
},
});
return artifact.id;
}
/** A device item in `space` that waits for its original. */
async function remoteAsset(space: Space, index: number): Promise<string> {
const source = await procedure(
rpc(actorOf(space)),
"sources/registerClient",
)({ kind: "ios_library", label: "Phone", deviceId: `device-${index}` });
const synced = await repos.upsertSourceItems(space.spaceId, (source as { id: string }).id, [
sourceItem(index),
]);
const assetId = synced.changedAssetIds[0] as string;
await repos.requestUpload(space.spaceId, [assetId]);
return assetId;
}
function putChunk(uploadId: string, index: number, body: BodyInit, actor: Actor | null) {
currentActor = actor;
return app.request(`/api/media/uploads/${uploadId}/chunks/${index}`, { method: "PUT", body });
}
beforeAll(async () => {
db = openMediaTestDb(databaseUrl!);
repos = createMediaRepos(db.prisma);
const root = await mkdtemp(path.join(tmpdir(), "media-api-pg-"));
mediaRoot = path.join(root, "media");
await mkdir(mediaRoot);
renderOutputs = new Map();
a = await db.newUser();
b = await db.newUser();
seededA = await seedMediaSpace(repos, a.spaceId, a.userId);
seededB = await seedMediaSpace(repos, b.spaceId, b.userId);
setup();
});
afterAll(async () => {
await db?.cleanup();
if (mediaRoot) await rm(path.dirname(mediaRoot), { recursive: true, force: true });
});
it("refuses another space's ids in every media procedure and leaves that space untouched", async () => {
const before = await snapshot(a.spaceId);
const foreignIds = Object.values(seededA);
const intruder = rpc(actorOf(b));
const inputs = sampleMediaInputs(seededA, testBrandKit);
for (const [rpcPath, sample] of Object.entries(inputs)) {
if (sample.foreign === "dedicated") continue;
const call = procedure(intruder, rpcPath)(sample.input);
if (sample.foreign === "not_found") {
expect(await rpcCode(call), rpcPath).toBe("NOT_FOUND");
continue;
}
const result = JSON.stringify(await call);
for (const id of foreignIds) expect(result, rpcPath).not.toContain(id);
}
expect(await snapshot(a.spaceId)).toBe(before);
// No service ever received the other space's ids.
const leaked = calls.filter((call) =>
foreignIds.some((id) => JSON.stringify(call.input ?? null).includes(id)),
);
expect(leaked).toEqual([]);
expect(calls.every((call) => call.actor.spaceId === b.spaceId)).toBe(true);
});
it("serves the same inputs inside the caller's own space", async () => {
// Positive control: NOT_FOUND above comes from isolation, not from bad samples.
const fabricated = new Set(["collections/merge", "people/merge"]);
for (const [rpcPath, sample] of Object.entries(sampleMediaInputs(seededA, testBrandKit))) {
if (sample.foreign !== "not_found" || fabricated.has(rpcPath)) continue;
const space = await db.newUser();
const seeded = await seedMediaSpace(repos, space.spaceId, space.userId);
const input = sampleMediaInputs(seeded, testBrandKit)[rpcPath]?.input;
const code = await rpcCode(procedure(rpc(actorOf(space)), rpcPath)(input));
expect(code, rpcPath).not.toBe("NOT_FOUND");
}
}, 120_000);
it("refuses another space's records referenced from the caller's own ones", async () => {
const own = rpc(actorOf(b));
const foreignArtifact = await artifactIn(a);
const attempts: Array<[string, unknown]> = [
["projects/update", { projectId: seededB.projectId, collectionIds: [seededA.collectionId] }],
["projects/update", { projectId: seededB.projectId, brandKitId: seededA.brandKitId }],
[
"projects/create",
{
title: "Mixed",
goal: "teaser",
assetIds: [seededA.assetId],
profileIds: ["example-short-vertical"],
},
],
["people/assignTrack", { trackId: seededB.trackId, personId: seededA.personId }],
[
"marks/create",
{
kind: "tattoo",
assetId: seededB.assetId,
personId: seededA.personId,
prompt: { tMs: 0 },
treatment: "blur",
},
],
["marks/update", { markId: seededB.markId, personId: seededA.personId }],
[
"assets/setPaidOnly",
{ segmentIds: [seededB.segmentId, seededA.segmentId], paidOnly: true },
],
["collections/merge", { collectionIds: [seededB.collectionId, seededA.collectionId] }],
["people/merge", { personIds: [seededB.personId, seededA.personId] }],
["brandKits/upsert", { kit: { ...testBrandKit, logoArtifactIds: [foreignArtifact] } }],
[
"people/recordConsent",
{
personId: seededB.personId,
consentArtifactId: foreignArtifact,
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "document_on_file",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
},
],
];
const before = await snapshot(a.spaceId);
for (const [rpcPath, input] of attempts) {
expect(await rpcCode(procedure(own, rpcPath)(input)), rpcPath).toBe("NOT_FOUND");
}
expect(await snapshot(a.spaceId)).toBe(before);
const person = await db.prisma.mediaPerson.findUniqueOrThrow({
where: { id: seededB.personId },
});
expect(person.consentRecordedAt).toBeNull();
});
it("records consent only as the signed-in member and ignores it on updates", async () => {
const space = await db.newUser();
const seeded = await seedMediaSpace(repos, space.spaceId, space.userId);
const client = rpc(actorOf(space));
await procedure(
client,
"people/update",
)({
personId: seeded.personId,
displayName: "Person 1",
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
});
let person = await db.prisma.mediaPerson.findUniqueOrThrow({ where: { id: seeded.personId } });
expect(person.consentRecordedAt).toBeNull();
expect(person.ageVerifiedAt).toBeNull();
const artifactId = await artifactIn(space);
await procedure(
client,
"people/recordConsent",
)({
personId: seeded.personId,
consentArtifactId: artifactId,
consentRecordedAt: "2026-09-01T10:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T10:00:00.000Z",
});
person = await db.prisma.mediaPerson.findUniqueOrThrow({ where: { id: seeded.personId } });
expect(person.consentRecordedBy).toBe(space.userId);
expect(person.consentArtifactId).toBe(artifactId);
expect(person.ageVerificationMethod).toBe("platform_verified");
});
it("assembles a resumable chunked upload and hands it to ingest", async () => {
const assetId = await remoteAsset(b, 101);
const bytes = new TextEncoder().encode("0123456789");
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: bytes.length,
sha256: sha256(bytes),
fileName: "clip.mp4",
chunkBytes: 4,
});
const owner = actorOf(b);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), actorOf(a))).status).toBe(404);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), null)).status).toBe(401);
expect((await putChunk(upload.id, 3, bytes.slice(0, 2), owner)).status).toBe(400);
expect((await putChunk(upload.id, 0, bytes.slice(0, 5), owner)).status).toBe(413);
expect((await putChunk(upload.id, 0, bytes.slice(0, 3), owner)).status).toBe(400);
// Out of order and repeated chunks are fine; the client resumes from status.
expect((await putChunk(upload.id, 2, bytes.slice(8), owner)).status).toBe(200);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(200);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(200);
const client = rpc(owner);
const partial = (await procedure(client, "uploads/status")({ uploadId: upload.id })) as {
receivedChunks: number[];
};
expect([...partial.receivedChunks].sort()).toEqual([0, 2]);
expect(await rpcCode(procedure(client, "uploads/complete")({ uploadId: upload.id }))).toBe(
"CONFLICT",
);
expect((await putChunk(upload.id, 1, bytes.slice(4, 8), owner)).status).toBe(200);
calls.length = 0;
const done = (await procedure(client, "uploads/complete")({ uploadId: upload.id })) as {
status: string;
};
expect(done.status).toBe("complete");
const handed = calls.filter((call) => call.method === "uploadCompleted");
expect(handed).toHaveLength(1);
const input = handed[0]?.input as { tempPath: string; sha256: string; assetId: string };
expect(input.assetId).toBe(assetId);
expect(input.sha256).toBe(sha256(bytes));
expect(input.tempPath.startsWith(path.join(mediaRoot, b.spaceId, "uploads"))).toBe(true);
expect(await readFile(input.tempPath, "utf8")).toBe("0123456789");
const asset = await repos.getAssetRecord(b.spaceId, assetId);
expect(asset.storage).toBe("uploaded");
expect(asset.storagePath).toBe(`${b.spaceId}/uploads/${upload.id}/file`);
// Completing again is idempotent and does not hand the file over twice.
await procedure(client, "uploads/complete")({ uploadId: upload.id });
expect(calls.filter((call) => call.method === "uploadCompleted")).toHaveLength(1);
expect((await putChunk(upload.id, 0, bytes.slice(0, 4), owner)).status).toBe(409);
});
it("opens uploads through the RPC only for items that wait for their original", async () => {
const client = rpc(actorOf(b));
const assetId = await remoteAsset(b, 102);
const input = { assetId, sizeBytes: 10, sha256: "a".repeat(64), fileName: "clip.mp4" };
const first = (await procedure(client, "uploads/begin")(input)) as { id: string };
const resumed = (await procedure(client, "uploads/begin")(input)) as { id: string };
expect(resumed.id).toBe(first.id);
const referenced = { ...input, assetId: seededB.assetId };
expect(await rpcCode(procedure(client, "uploads/begin")(referenced))).toBe("CONFLICT");
const tooLarge = { ...input, sizeBytes: 21 * 1024 * 1024 * 1024 };
expect(await rpcCode(procedure(client, "uploads/begin")(tooLarge))).toBe("BAD_REQUEST");
});
it("fails an upload whose bytes do not match the announced checksum", async () => {
const assetId = await remoteAsset(b, 103);
const bytes = new TextEncoder().encode("abcdef");
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: bytes.length,
sha256: "f".repeat(64),
fileName: "clip.mp4",
chunkBytes: 4,
});
const owner = actorOf(b);
await putChunk(upload.id, 0, bytes.slice(0, 4), owner);
await putChunk(upload.id, 1, bytes.slice(4), owner);
const client = rpc(owner);
expect(await rpcCode(procedure(client, "uploads/complete")({ uploadId: upload.id }))).toBe(
"BAD_REQUEST",
);
const status = (await procedure(client, "uploads/status")({ uploadId: upload.id })) as {
status: string;
};
expect(status.status).toBe("failed");
const dir = path.join(mediaRoot, b.spaceId, "uploads", upload.id);
await expect(stat(dir)).rejects.toMatchObject({ code: "ENOENT" });
});
it("expires an idle upload and starts a fresh one", async () => {
const assetId = await remoteAsset(b, 104);
const input = { assetId, sizeBytes: 10, sha256: "b".repeat(64), fileName: "clip.mp4" };
const first = (await procedure(rpc(actorOf(b)), "uploads/begin")(input)) as { id: string };
setup({ now: () => new Date(Date.now() + MEDIA_UPLOAD_IDLE_MS + 60_000) });
try {
const response = await putChunk(first.id, 0, new Uint8Array(10), actorOf(b));
expect(response.status).toBe(409);
const status = (await procedure(
rpc(actorOf(b)),
"uploads/status",
)({
uploadId: first.id,
})) as { status: string };
expect(status.status).toBe("failed");
const fresh = (await procedure(rpc(actorOf(b)), "uploads/begin")(input)) as { id: string };
expect(fresh.id).not.toBe(first.id);
} finally {
setup();
}
});
it("drops a chunk that does not arrive in time", async () => {
const assetId = await remoteAsset(b, 105);
const upload = await repos.beginUpload(b.spaceId, {
assetId,
userId: b.userId,
sizeBytes: 10,
sha256: "c".repeat(64),
fileName: "clip.mp4",
});
setup({ chunkTimeoutMs: 50 });
try {
const stalled = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array(2));
},
});
currentActor = actorOf(b);
const response = await app.request(`/api/media/uploads/${upload.id}/chunks/0`, {
method: "PUT",
body: stalled,
duplex: "half",
} as RequestInit);
expect(response.status).toBe(408);
const status = await repos.uploadStatus(b.spaceId, upload.id);
expect(status.receivedChunks).toEqual([]);
} finally {
setup();
}
});
it("streams render files only inside the caller's space and render folder", async () => {
const renderDir = path.join(mediaRoot, a.spaceId, "renders", seededA.renderId);
await mkdir(renderDir, { recursive: true });
const video = path.join(renderDir, "video.mp4");
await writeFile(video, "0123456789");
renderOutputs.set(seededA.renderId, video);
const url = `/api/media/renders/${seededA.renderId}/files/video.mp4`;
currentActor = actorOf(a);
const full = await app.request(url);
expect(full.status).toBe(200);
expect(await full.text()).toBe("0123456789");
const ranged = await app.request(url, { headers: { range: "bytes=2-4" } });
expect(ranged.status).toBe(206);
expect(ranged.headers.get("content-range")).toBe("bytes 2-4/10");
expect(await ranged.text()).toBe("234");
const download = await app.request(`${url}?download=1`);
expect(download.headers.get("content-disposition")).toContain("attachment");
// A render that has not passed its gate never leaves as a download.
const queuedVideo = path.join(mediaRoot, a.spaceId, "renders", seededA.queuedRenderId, "v.mp4");
await mkdir(path.dirname(queuedVideo), { recursive: true });
await writeFile(queuedVideo, "x");
renderOutputs.set(seededA.queuedRenderId, queuedVideo);
const queuedUrl = `/api/media/renders/${seededA.queuedRenderId}/files/v.mp4`;
expect((await app.request(queuedUrl)).status).toBe(200);
expect((await app.request(`${queuedUrl}?download=1`)).status).toBe(404);
// A path the service reports outside the render folder is refused.
const outside = path.join(mediaRoot, a.spaceId, "video.mp4");
await writeFile(outside, "secret");
renderOutputs.set(seededA.renderId, outside);
expect((await app.request(url)).status).toBe(404);
renderOutputs.set(seededA.renderId, video);
currentActor = actorOf(b);
expect((await app.request(url)).status).toBe(404);
expect((await app.request(`/api/media/assets/${seededA.assetId}/thumb`)).status).toBe(404);
expect((await app.request(`/api/media/bundles/${seededA.publicationId}`)).status).toBe(404);
});
it("refuses a folder source to members who do not own the deployment", async () => {
const folder = await mkdtemp(path.join(tmpdir(), "media-footage-"));
try {
const input = { label: "Footage", path: folder };
const member = procedure(rpc(actorOf(b)), "sources/createFolder")(input);
expect(await rpcCode(member)).toBe("FORBIDDEN");
const created = (await procedure(rpc(actorOf(b, true)), "sources/createFolder")(input)) as {
id: string;
kind: string;
};
expect(created.kind).toBe("folder");
const row = await db.prisma.mediaSource.findUniqueOrThrow({ where: { id: created.id } });
expect(row.ownerUserId).toBe(b.userId);
} finally {
await rm(folder, { recursive: true, force: true });
}
});
});

333
apps/api/src/media.test.ts Normal file
View File

@ -0,0 +1,333 @@
import { mkdir, mkdtemp, rm, symlink } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createRouterClient, ORPCError } from "@orpc/server";
import type { Actor } from "@rakazo/contracts";
import { MediaToolInputSchemas, mediaRpcShapes } from "@rakazo/contracts";
import type { MediaRepos, PrismaClient } from "@rakazo/db";
import { InvalidMediaInputError, IsolationError, MediaStateConflictError } from "@rakazo/db";
import { PathOutsideRootError } from "@rakazo/media/node";
import { Hono } from "hono";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
createMediaRouter,
MEDIA_NOT_CONFIGURED,
mediaRpcError,
unconfiguredCapabilities,
validateMediaFolder,
} from "./media.js";
import { mountMediaHttpRoutes, parseByteRange } from "./media-http.js";
import type { MediaSampleIds } from "./media-test-fakes.js";
import { fakeMediaServices, mediaProcedurePaths, sampleMediaInputs } from "./media-test-fakes.js";
import { expectedChunkBytes, uploadChunkCount } from "./media-uploads.js";
const member: Actor = {
userId: "user-member",
spaceId: "space-a",
email: "member@rakazo.test",
isDeploymentOwner: false,
};
const owner: Actor = { ...member, userId: "user-owner", isDeploymentOwner: true };
const ids = Object.fromEntries(
[
"sourceId",
"assetId",
"secondAssetId",
"segmentId",
"trackId",
"personId",
"markId",
"collectionId",
"brandKitId",
"projectId",
"planId",
"renderId",
"publicationId",
"mappingId",
"uploadId",
].map((key) => [key, `id-${key.toLowerCase()}`]),
) as unknown as MediaSampleIds;
const kit = {
name: "Kit",
colors: {
primary: "#101010",
secondary: "#202020",
accent: "#303030",
text: "#f0f0f0",
background: "#000000",
},
};
function client(router: ReturnType<typeof createMediaRouter>, actor: Actor | null) {
return createRouterClient(router, { context: { actor } });
}
function procedure(target: unknown, rpcPath: string) {
let node: unknown = target;
for (const key of rpcPath.split("/")) node = (node as Record<string, unknown>)[key];
return node as (input: unknown) => Promise<unknown>;
}
async function rpcCode(promise: Promise<unknown>): Promise<string> {
try {
await promise;
} catch (error) {
if (error instanceof ORPCError) return error.code;
throw error;
}
return "OK";
}
/** Repositories that fail the test when reached. */
function untouchedRepos(): MediaRepos {
return new Proxy({} as MediaRepos, {
get: (_target, key) => () => {
throw new Error(`repository ${String(key)} must not be called`);
},
});
}
const prismaStub = {} as PrismaClient;
describe("media router without a configured sidecar", () => {
const router = createMediaRouter({ prisma: prismaStub, services: null, dataDir: "/data" });
const inputs = sampleMediaInputs(ids, kit);
it("has a sample input for every media procedure", () => {
expect(Object.keys(inputs).sort()).toEqual(mediaProcedurePaths().sort());
});
it("reports media as not configured so every surface keeps it hidden", async () => {
const capabilities = await client(router, member).capabilities({});
expect(capabilities).toEqual(unconfiguredCapabilities());
expect(capabilities.configured).toBe(false);
expect(
Object.values(capabilities.operations).every(
(op) => !(op as { available: boolean }).available,
),
).toBe(true);
});
it("refuses every other procedure", async () => {
for (const [rpcPath, sample] of Object.entries(inputs)) {
if (rpcPath === "capabilities") continue;
const promise = procedure(client(router, member), rpcPath)(sample.input);
await expect(promise, rpcPath).rejects.toMatchObject({
code: "PRECONDITION_FAILED",
message: MEDIA_NOT_CONFIGURED,
});
}
});
it("requires a signed-in member for every procedure", async () => {
for (const [rpcPath, sample] of Object.entries(inputs)) {
const code = await rpcCode(procedure(client(router, null), rpcPath)(sample.input));
expect(code, rpcPath).toBe("UNAUTHORIZED");
}
});
it("answers the byte routes with 404 and anonymous callers with 401", async () => {
const app = new Hono();
let actor: Actor | null = member;
mountMediaHttpRoutes(app, { services: null }, async () => actor);
const requests = [
new Request("http://api/api/media/uploads/u1/chunks/0", { method: "PUT", body: "abcd" }),
new Request("http://api/api/media/renders/r1/files/video.mp4"),
new Request("http://api/api/media/assets/a1/thumb"),
new Request("http://api/api/media/assets/a1/proxy"),
new Request("http://api/api/media/bundles/p1"),
];
for (const request of requests) {
expect((await app.request(request.clone())).status, request.url).toBe(404);
}
actor = null;
for (const request of requests) {
expect((await app.request(request.clone())).status, request.url).toBe(401);
}
});
});
describe("media router permissions", () => {
let root: string;
beforeAll(async () => {
root = await mkdtemp(path.join(tmpdir(), "media-api-"));
});
afterAll(async () => {
await rm(root, { recursive: true, force: true });
});
it("lets only the deployment owner add a folder source", async () => {
const created: unknown[] = [];
const repos = new Proxy({} as MediaRepos, {
get: (_target, key) => {
if (key === "createSource") {
return async (_spaceId: string, input: unknown) => {
created.push(input);
return {
id: "src-1",
kind: "folder",
label: "Footage",
status: "idle",
assetCount: 0,
lastSyncedAt: null,
error: null,
createdAt: "2026-10-01T00:00:00.000Z",
};
};
}
return () => {
throw new Error(`repository ${String(key)} must not be called`);
};
},
});
const { services, calls } = fakeMediaServices({ repos, mediaRoot: path.join(root, "media") });
const footage = path.join(root, "footage");
await mkdir(footage);
const router = createMediaRouter({
prisma: prismaStub,
services,
dataDir: path.join(root, "data"),
appRoot: path.join(root, "app"),
});
const denied = client(router, member).sources.createFolder({ label: "Footage", path: footage });
await expect(denied).rejects.toMatchObject({ code: "FORBIDDEN" });
expect(created).toEqual([]);
expect(calls).toEqual([]);
await client(router, owner).sources.createFolder({ label: "Footage", path: footage });
expect(created).toEqual([
{ kind: "folder", label: "Footage", ownerUserId: owner.userId, config: { path: footage } },
]);
expect(calls.map((call) => `${call.service}.${call.method}`)).toEqual(["ingest.requestSync"]);
});
it("accepts only a real, specific folder apart from evOo's own directories", async () => {
const dataDir = path.join(root, "data");
const appRoot = path.join(root, "app");
const folder = path.join(root, "clips");
await mkdir(path.join(dataDir, "media"), { recursive: true });
await mkdir(path.join(appRoot, "packages"), { recursive: true });
await mkdir(folder, { recursive: true });
await symlink(folder, path.join(root, "clips-link"));
const forbidden = { dataDir, appRoot };
await expect(validateMediaFolder(folder, forbidden)).resolves.toBe(folder);
const rejected = [
"clips",
path.join(root, "missing"),
path.join(root, "clips-link"),
"/",
dataDir,
path.join(dataDir, "media"),
root,
path.join(appRoot, "packages"),
];
for (const candidate of rejected) {
await expect(validateMediaFolder(candidate, forbidden), candidate).rejects.toBeInstanceOf(
InvalidMediaInputError,
);
}
});
});
describe("consent stays with the signed-in user", () => {
it("never accepts consent or age fields from the agent's people tool", () => {
const schema = MediaToolInputSchemas.media_people;
expect(schema.shape.action.options).toEqual(["list", "update", "merge"]);
const parsed = schema.parse({
action: "update",
personId: "person-1",
consentArtifactId: "artifact-1",
consentRecordedAt: "2026-09-01T00:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T00:00:00.000Z",
});
expect(Object.keys(parsed).sort()).toEqual(["action", "personId"]);
});
it("drops consent fields sent to people.update", async () => {
const patches: unknown[] = [];
const repos = new Proxy({} as MediaRepos, {
get: (_target, key) => {
if (key === "updatePerson") {
return async (_spaceId: string, _personId: string, patch: unknown) => {
patches.push(patch);
return {};
};
}
return () => {
throw new Error(`repository ${String(key)} must not be called`);
};
},
});
const { services } = fakeMediaServices({ repos, mediaRoot: "/unused" });
const router = createMediaRouter({ prisma: prismaStub, services, dataDir: "/data" });
await procedure(
client(router, member),
"people/update",
)({
personId: "person-1",
displayName: "Person 1",
consentRecordedAt: "2026-09-01T00:00:00.000Z",
ageVerificationMethod: "platform_verified",
ageVerifiedAt: "2026-09-01T00:00:00.000Z",
}).catch(() => undefined);
expect(patches).toHaveLength(1);
expect(Object.entries(patches[0] as object).filter(([, value]) => value !== undefined)).toEqual(
[["displayName", "Person 1"]],
);
});
it("marks recordConsent as the only user-only write and keeps it out of the services", () => {
const people = mediaRpcShapes.people as Record<string, unknown>;
expect(Object.keys(people)).toContain("recordConsent");
const { services } = fakeMediaServices({ repos: untouchedRepos(), mediaRoot: "/unused" });
const serviceMethods = (["ingest", "privacy", "render", "marketing"] as const).flatMap((name) =>
Object.keys(services[name] as unknown as Record<string, unknown>),
);
expect(serviceMethods.filter((method) => /consent|age/i.test(method))).toEqual([]);
});
});
describe("media error mapping", () => {
it("never reveals whether a foreign id exists", () => {
expect((mediaRpcError(new IsolationError()) as ORPCError<string, unknown>).code).toBe(
"NOT_FOUND",
);
expect(
(mediaRpcError(new PathOutsideRootError("../x")) as ORPCError<string, unknown>).code,
).toBe("BAD_REQUEST");
expect(
(mediaRpcError(new MediaStateConflictError("busy")) as ORPCError<string, unknown>).code,
).toBe("CONFLICT");
const other = new Error("boom");
expect(mediaRpcError(other)).toBe(other);
});
});
describe("upload and streaming helpers", () => {
it("splits uploads into full chunks plus a shorter last one", () => {
const row = { sizeBytes: 10n, chunkBytes: 4 };
expect(uploadChunkCount(row)).toBe(3);
expect([0, 1, 2].map((index) => expectedChunkBytes(row, index))).toEqual([4, 4, 2]);
expect(uploadChunkCount({ sizeBytes: 8n, chunkBytes: 4 })).toBe(2);
expect(expectedChunkBytes({ sizeBytes: 8n, chunkBytes: 4 }, 1)).toBe(4);
});
it("parses single byte ranges and refuses unsatisfiable ones", () => {
expect(parseByteRange(undefined, 100)).toBeNull();
expect(parseByteRange("bytes=0-9", 100)).toEqual({ start: 0, end: 9 });
expect(parseByteRange("bytes=90-", 100)).toEqual({ start: 90, end: 99 });
expect(parseByteRange("bytes=-10", 100)).toEqual({ start: 90, end: 99 });
expect(parseByteRange("bytes=50-500", 100)).toEqual({ start: 50, end: 99 });
expect(parseByteRange("bytes=0-1,5-6", 100)).toBeNull();
expect(parseByteRange("bytes=100-", 100)).toBe("unsatisfiable");
expect(parseByteRange("bytes=9-3", 100)).toBe("unsatisfiable");
expect(parseByteRange("items=0-1", 100)).toBe("unsatisfiable");
expect(parseByteRange("bytes=-0", 100)).toBe("unsatisfiable");
});
});

552
apps/api/src/media.ts Normal file
View File

@ -0,0 +1,552 @@
import { lstat, realpath, rm } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { implement, ORPCError } from "@orpc/server";
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import type { MediaServices } from "@rakazo/adapters";
import { recordMetric, resetTeaserModel, saveBrandKit, segmentFunnel } from "@rakazo/adapters";
import type { Actor, BrandKit, MediaCapabilities, MediaSourceKind } from "@rakazo/contracts";
import { appContract, MediaInferenceOperationSchema } from "@rakazo/contracts";
import type { PrismaClient } from "@rakazo/db";
import {
InvalidMediaInputError,
InvalidProjectWorkspacePathError,
IsolationError,
lockMediaKey,
MediaStateConflictError,
mapMediaSource,
resolveProjectWorkspaceDirectory,
} from "@rakazo/db";
import { summarizeProfile } from "@rakazo/media";
import { PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import { beginMediaUpload, completeMediaUpload, UploadChunkError } from "./media-uploads.js";
export interface MediaRouterDeps {
prisma: PrismaClient;
/** Null when Media Studio is not configured: everything but capabilities refuses. */
services?: MediaServices | null;
/** DATA_DIR; a folder source may neither sit inside it nor contain it. */
dataDir: string;
/** Application install directory a folder source may not point into. */
appRoot?: string;
}
export const MEDIA_NOT_CONFIGURED = "Media Studio is not configured";
const DEFAULT_APP_ROOT = fileURLToPath(new URL("../../../", import.meta.url));
/** Translate domain errors into RPC errors; everything else passes through. */
export function mediaRpcError(error: unknown): unknown {
if (error instanceof ORPCError) return error;
if (error instanceof IsolationError) return new ORPCError("NOT_FOUND");
if (
error instanceof InvalidMediaInputError ||
error instanceof InvalidProjectWorkspacePathError
) {
return new ORPCError("BAD_REQUEST", { message: error.message });
}
if (error instanceof PathOutsideRootError) {
return new ORPCError("BAD_REQUEST", { message: "That path is not allowed." });
}
if (error instanceof MediaStateConflictError) {
return new ORPCError("CONFLICT", { message: error.message });
}
if (error instanceof UploadChunkError) {
return new ORPCError(error.status === 409 ? "CONFLICT" : "BAD_REQUEST", {
message: error.message,
});
}
if (error instanceof UnsupportedMediaOperationError) {
return new ORPCError("PRECONDITION_FAILED", { message: error.message });
}
return error;
}
export function unconfiguredCapabilities(): MediaCapabilities {
const unavailable = {
available: false,
experimental: false,
model: null,
license: null,
commercialUse: null,
};
return {
configured: false,
operations: Object.fromEntries(
MediaInferenceOperationSchema.options.map((operation) => [operation, unavailable]),
) as MediaCapabilities["operations"],
accelerator: "none",
hardwareEncoders: [],
};
}
function overlaps(a: string, b: string): boolean {
const inside = (root: string, candidate: string) => {
const relative = path.relative(root, candidate);
return relative === "" || (!relative.startsWith("..") && !path.isAbsolute(relative));
};
return inside(a, b) || inside(b, a);
}
async function realOrResolved(target: string): Promise<string> {
return realpath(target).catch(() => path.resolve(target));
}
/**
* A host folder for a folder source: absolute, an existing directory, not a
* symlink, and apart from DATA_DIR and the application itself. Returns the real path.
*/
export async function validateMediaFolder(
input: string,
forbidden: { dataDir: string; appRoot: string },
): Promise<string> {
const trimmed = input.trim();
if (!path.isAbsolute(trimmed)) {
throw new InvalidMediaInputError("Use an absolute folder path.");
}
const info = await lstat(trimmed).catch(() => null);
if (!info) throw new InvalidMediaInputError("That folder does not exist or is not readable.");
if (info.isSymbolicLink())
throw new InvalidMediaInputError("Choose the folder, not a link to it.");
const resolved = await resolveProjectWorkspaceDirectory(trimmed);
if (path.parse(resolved).root === resolved) {
throw new InvalidMediaInputError("Choose a specific folder.");
}
const dataDir = await realOrResolved(forbidden.dataDir);
const appRoot = await realOrResolved(forbidden.appRoot);
const relativeToApp = path.relative(appRoot, resolved);
const insideApp =
relativeToApp === "" || (!relativeToApp.startsWith("..") && !path.isAbsolute(relativeToApp));
if (overlaps(dataDir, resolved) || insideApp) {
throw new InvalidMediaInputError("That folder belongs to evOo itself.");
}
return resolved;
}
/** Artifacts referenced from media records must belong to the caller's space. */
export async function assertSpaceArtifacts(
prisma: PrismaClient,
spaceId: string,
artifactIds: string[],
): Promise<void> {
const ids = [...new Set(artifactIds)];
if (ids.length === 0) return;
const owned = await prisma.artifact.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
}
function brandKitArtifactIds(kit: BrandKit): string[] {
return [
...kit.fonts.map((font) => font.artifactId),
...kit.logoArtifactIds,
...(kit.lutArtifactId ? [kit.lutArtifactId] : []),
...kit.musicBeds.map((bed) => bed.artifactId),
];
}
async function removeMediaPath(mediaRoot: string, relative: string | null): Promise<void> {
if (!relative) return;
await rm(await resolveInside(mediaRoot, relative), { recursive: true, force: true });
}
export function createMediaRouter(deps: MediaRouterDeps) {
const os = implement(appContract.media).$context<{ actor: Actor | null; signal?: AbortSignal }>();
const authed = os.use(async ({ context, next }) => {
if (!context.actor) throw new ORPCError("UNAUTHORIZED");
try {
return await next({ context: { ...context, actor: context.actor } });
} catch (error) {
throw mediaRpcError(error);
}
});
const media = authed.use(async ({ next }) => {
if (!deps.services) {
throw new ORPCError("PRECONDITION_FAILED", { message: MEDIA_NOT_CONFIGURED });
}
return next({ context: { services: deps.services } });
});
const forbiddenRoots = { dataDir: deps.dataDir, appRoot: deps.appRoot ?? DEFAULT_APP_ROOT };
async function knownProfiles(services: MediaServices, profileIds: string[] | undefined) {
if (!profileIds) return;
const { profiles } = await services.deps.profiles();
const unknown = profileIds.filter((id) => !profiles.has(id));
if (unknown.length > 0) {
throw new InvalidMediaInputError(`Unknown profile: ${unknown.join(", ")}`);
}
}
return os.router({
capabilities: authed.capabilities.handler(async ({ context }) => {
const services = deps.services;
if (!services) return unconfiguredCapabilities();
const hardwareEncoders = await services.deps.renderer.hardwareEncoders().catch(() => []);
const inference = services.deps.inference;
if (!inference) return { ...unconfiguredCapabilities(), hardwareEncoders };
const capabilities = await inference
.capabilities(context.signal)
.catch(() => ({ ...unconfiguredCapabilities(), configured: true }));
return { ...capabilities, configured: true, hardwareEncoders };
}),
sources: {
list: media.sources.list.handler(({ context }) =>
context.services.deps.repos.listSources(context.actor.spaceId),
),
createFolder: media.sources.createFolder.handler(async ({ context, input }) => {
// A folder lives on the host, and the host is the deployment owner's machine.
if (!context.actor.isDeploymentOwner) throw new ORPCError("FORBIDDEN");
const folder = await validateMediaFolder(input.path, forbiddenRoots);
const { repos } = context.services.deps;
const source = await repos.createSource(context.actor.spaceId, {
kind: "folder",
label: input.label,
ownerUserId: context.actor.userId,
config: { path: folder },
});
await context.services.ingest.requestSync(context.actor, source.id);
return source;
}),
registerClient: media.sources.registerClient.handler(async ({ context, input }) => {
const { spaceId, userId } = context.actor;
const kind: MediaSourceKind = input.kind;
// One source per device: re-registering returns the existing one.
return deps.prisma.$transaction(async (tx) => {
await lockMediaKey(tx, `source:${spaceId}:${kind}:${input.deviceId}`);
const existing = await tx.mediaSource.findFirst({
where: { spaceId, kind, config: { path: ["deviceId"], equals: input.deviceId } },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
include: { _count: { select: { assets: true } } },
});
if (existing) return mapMediaSource(existing, existing._count.assets);
const created = await tx.mediaSource.create({
data: {
spaceId,
kind,
label: input.label,
ownerUserId: userId,
config: { deviceId: input.deviceId },
},
});
return mapMediaSource(created, 0);
});
}),
pushItems: media.sources.pushItems.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
return context.services.ingest.pushItems(context.actor, input);
}),
sync: media.sources.sync.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
await context.services.ingest.requestSync(context.actor, input.sourceId);
return { ok: true as const };
}),
remove: media.sources.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.getSource(context.actor.spaceId, input.sourceId);
await context.services.ingest.removeSource(context.actor, input.sourceId);
return { ok: true as const };
}),
},
assets: {
search: media.assets.search.handler(({ context, input }) =>
context.services.deps.repos.searchAssets(context.actor.spaceId, input),
),
get: media.assets.get.handler(({ context, input }) =>
context.services.deps.repos.getAsset(context.actor.spaceId, input.assetId),
),
setPaidOnly: media.assets.setPaidOnly.handler(async ({ context, input }) => {
await context.services.deps.repos.setSegmentsPaidOnly(
context.actor.spaceId,
input.segmentIds,
input.paidOnly,
);
return { ok: true as const };
}),
requestUpload: media.assets.requestUpload.handler(async ({ context, input }) => ({
awaiting: await context.services.deps.repos.requestUpload(
context.actor.spaceId,
input.assetIds,
),
})),
},
uploads: {
begin: media.uploads.begin.handler(({ context, input }) =>
beginMediaUpload(context.services, context.actor, input),
),
status: media.uploads.status.handler(({ context, input }) =>
context.services.deps.repos.uploadStatus(context.actor.spaceId, input.uploadId),
),
complete: media.uploads.complete.handler(({ context, input }) =>
completeMediaUpload(context.services, context.actor, input.uploadId),
),
},
collections: {
list: media.collections.list.handler(({ context, input }) =>
context.services.deps.repos.listCollections(context.actor.spaceId, input),
),
get: media.collections.get.handler(({ context, input }) =>
context.services.deps.repos.getCollection(context.actor.spaceId, input.collectionId),
),
rename: media.collections.rename.handler(({ context, input }) =>
context.services.deps.repos.renameCollection(
context.actor.spaceId,
input.collectionId,
input.title,
),
),
merge: media.collections.merge.handler(({ context, input }) =>
context.services.deps.repos.mergeCollections(context.actor.spaceId, input.collectionIds),
),
split: media.collections.split.handler(({ context, input }) =>
context.services.deps.repos.splitCollection(
context.actor.spaceId,
input.collectionId,
input.atAssetId,
),
),
createManual: media.collections.createManual.handler(({ context, input }) =>
context.services.deps.repos.createManualCollection(
context.actor.spaceId,
input.title,
input.assetIds,
),
),
recluster: media.collections.recluster.handler(async ({ context }) => {
await context.services.ingest.recluster(context.actor);
return { ok: true as const };
}),
},
people: {
list: media.people.list.handler(({ context }) =>
context.services.deps.repos.listPeople(context.actor.spaceId),
),
update: media.people.update.handler(({ context, input }) =>
context.services.deps.repos.updatePerson(context.actor.spaceId, input.personId, {
displayName: input.displayName,
isSelf: input.isSelf,
confirmed: input.confirmed,
anonymizeByDefault: input.anonymizeByDefault,
}),
),
merge: media.people.merge.handler(({ context, input }) =>
context.services.deps.repos.mergePeople(context.actor.spaceId, input.personIds),
),
assignTrack: media.people.assignTrack.handler(({ context, input }) =>
context.services.deps.repos.assignTrack(
context.actor.spaceId,
input.trackId,
input.personId,
),
),
// The only consent write path: the signed-in member is the recorder.
recordConsent: media.people.recordConsent.handler(async ({ context, input }) => {
await assertSpaceArtifacts(
deps.prisma,
context.actor.spaceId,
input.consentArtifactId ? [input.consentArtifactId] : [],
);
return context.services.deps.repos.recordConsent(
{ spaceId: context.actor.spaceId, recordedByUserId: context.actor.userId },
input,
);
}),
remove: media.people.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.getPerson(context.actor.spaceId, input.personId);
await context.services.privacy.removePerson(context.actor, input.personId);
return { ok: true as const };
}),
},
marks: {
list: media.marks.list.handler(({ context, input }) =>
context.services.deps.repos.listMarks(context.actor.spaceId, {
...(input.personId ? { personId: input.personId } : {}),
...(input.assetId ? { assetId: input.assetId } : {}),
}),
),
create: media.marks.create.handler(async ({ context, input }) => {
const { repos } = context.services.deps;
await repos.getAssetRecord(context.actor.spaceId, input.assetId);
if (input.personId) await repos.getPerson(context.actor.spaceId, input.personId);
return context.services.privacy.createMark(context.actor, input);
}),
update: media.marks.update.handler(async ({ context, input }) => {
const { repos } = context.services.deps;
await repos.getMark(context.actor.spaceId, input.markId);
if (input.personId) await repos.getPerson(context.actor.spaceId, input.personId);
return context.services.privacy.updateMark(context.actor, input);
}),
remove: media.marks.remove.handler(async ({ context, input }) => {
const { repos, mediaRoot } = context.services.deps;
const { maskDir } = await repos.removeMark(context.actor.spaceId, input.markId);
await removeMediaPath(mediaRoot, maskDir);
return { ok: true as const };
}),
},
brandKits: {
list: media.brandKits.list.handler(({ context }) =>
context.services.deps.repos.listBrandKits(context.actor.spaceId),
),
upsert: media.brandKits.upsert.handler(async ({ context, input }) => {
await assertSpaceArtifacts(
deps.prisma,
context.actor.spaceId,
brandKitArtifactIds(input.kit),
);
return saveBrandKit(context.services.deps, context.actor, input);
}),
remove: media.brandKits.remove.handler(async ({ context, input }) => {
await context.services.deps.repos.removeBrandKit(context.actor.spaceId, input.id);
return { ok: true as const };
}),
templates: media.brandKits.templates.handler(({ context }) =>
context.services.deps.motion.templates(),
),
introVariants: media.brandKits.introVariants.handler(async ({ context, input }) => {
await context.services.deps.repos.getBrandKit(context.actor.spaceId, input.id);
return context.services.marketing.introVariants(context.actor, {
brandKitId: input.id,
count: input.count,
});
}),
},
profiles: {
list: media.profiles.list.handler(async ({ context }) => {
const { profiles } = await context.services.deps.profiles();
const now = context.services.deps.now();
return [...profiles.values()].map(({ profile, origin }) =>
summarizeProfile(profile, origin, now),
);
}),
get: media.profiles.get.handler(async ({ context, input }) => {
const { profiles } = await context.services.deps.profiles();
const entry = profiles.get(input.profileId);
if (!entry) throw new ORPCError("NOT_FOUND");
return entry.profile;
}),
},
projects: {
list: media.projects.list.handler(({ context, input }) =>
context.services.deps.repos.listProjects(context.actor.spaceId, input),
),
get: media.projects.get.handler(({ context, input }) =>
context.services.deps.repos.getProject(context.actor.spaceId, input.projectId),
),
create: media.projects.create.handler(async ({ context, input }) => {
await knownProfiles(context.services, input.profileIds);
return context.services.deps.repos.createProject(
context.actor.spaceId,
context.actor.userId,
input,
);
}),
update: media.projects.update.handler(async ({ context, input }) => {
await knownProfiles(context.services, input.profileIds);
const { projectId, ...patch } = input;
return context.services.deps.repos.updateProject(context.actor.spaceId, projectId, patch);
}),
remove: media.projects.remove.handler(async ({ context, input }) => {
const { repos, mediaRoot } = context.services.deps;
const { spaceId } = context.actor;
await repos.getProjectRecord(spaceId, input.projectId);
const renders = await repos.listRenders(spaceId, input.projectId);
await repos.removeProject(spaceId, input.projectId);
for (const render of renders)
await removeMediaPath(mediaRoot, `${spaceId}/renders/${render.id}`);
return { ok: true as const };
}),
autoPlan: media.projects.autoPlan.handler(async ({ context, input }) => {
await context.services.deps.repos.getProjectRecord(context.actor.spaceId, input.projectId);
await knownProfiles(context.services, input.profileIds);
return context.services.marketing.autoPlan(context.actor, input);
}),
},
plans: {
submit: media.plans.submit.handler(async ({ context, input }) => {
await context.services.deps.repos.getProjectRecord(context.actor.spaceId, input.projectId);
return context.services.marketing.submitPlan(context.actor, {
projectId: input.projectId,
edl: input.edl,
rationale: input.rationale,
createdBy: "user",
});
}),
},
renders: {
start: media.renders.start.handler(async ({ context, input }) => {
await context.services.deps.repos.getPlan(context.actor.spaceId, input.planId);
return context.services.render.start(context.actor, input);
}),
get: media.renders.get.handler(({ context, input }) =>
context.services.deps.repos.getRender(context.actor.spaceId, input.renderId),
),
cancel: media.renders.cancel.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.render.cancel(context.actor, input.renderId);
}),
thumbnails: media.renders.thumbnails.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.thumbnails(context.actor, input);
}),
},
marketing: {
copy: media.marketing.copy.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.copy(context.actor, input);
}),
},
publications: {
list: media.publications.list.handler(({ context, input }) =>
context.services.deps.repos.listPublications(context.actor.spaceId, input),
),
create: media.publications.create.handler(async ({ context, input }) => {
await context.services.deps.repos.getRender(context.actor.spaceId, input.renderId);
return context.services.marketing.createPublication(context.actor, input);
}),
bundle: media.publications.bundle.handler(async ({ context, input }) => {
await context.services.deps.repos.getPublication(
context.actor.spaceId,
input.publicationId,
);
return context.services.marketing.bundle(context.actor, input.publicationId);
}),
markPublished: media.publications.markPublished.handler(({ context, input }) =>
context.services.deps.repos.markPublicationPublished(
context.actor.spaceId,
input.publicationId,
input.externalUrl,
context.services.deps.now(),
),
),
cancel: media.publications.cancel.handler(async ({ context, input }) => {
await context.services.deps.repos.getPublication(
context.actor.spaceId,
input.publicationId,
);
return context.services.marketing.cancelPublication(context.actor, input.publicationId);
}),
},
metrics: {
mappings: media.metrics.mappings.handler(({ context }) =>
context.services.deps.repos.listMetricMappings(context.actor.spaceId),
),
saveMapping: media.metrics.saveMapping.handler(({ context, input }) =>
context.services.deps.repos.saveMetricMapping(context.actor.spaceId, input),
),
importCsv: media.metrics.importCsv.handler(async ({ context, input }) => {
await context.services.deps.repos.getMetricMapping(context.actor.spaceId, input.mappingId);
return context.services.marketing.importMetricsCsv(context.actor, input);
}),
record: media.metrics.record.handler(({ context, input }) =>
recordMetric(context.services.deps, context.actor, input),
),
funnel: media.metrics.funnel.handler(({ context, input }) =>
context.services.deps.repos.funnel(context.actor.spaceId, input.projectId),
),
segments: media.metrics.segments.handler(({ context, input }) =>
segmentFunnel(context.services.deps, context.actor, input.projectId),
),
resetTeaserModel: media.metrics.resetTeaserModel.handler(async ({ context }) => {
await resetTeaserModel(context.services.deps, context.actor);
return { ok: true as const };
}),
},
});
}

View File

@ -1,11 +1,42 @@
import { mkdir, mkdtemp, writeFile } from "node:fs/promises";
import { mkdir, mkdtemp, realpath, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type * as db from "@rakazo/db";
import { SpaceLimitError } from "@rakazo/db";
import { afterEach, describe, expect, it, vi } from "vitest";
import { getScanStatus, startOnboardingScan } from "./onboarding-scan.js";
const owner = { userId: "user-owner", isDeploymentOwner: true };
const member = { userId: "user-member", isDeploymentOwner: false };
const createSpaceForMember = vi.hoisted(() => vi.fn());
vi.mock("@rakazo/db", async (original) => ({
...(await original<typeof db>()),
createSpaceForMember,
}));
const owner = { userId: "user-owner", spaceId: "space-owner", isDeploymentOwner: true };
const member = { userId: "user-member", spaceId: "space-member", isDeploymentOwner: false };
type SpaceUpdate = { where: { id: string }; data: { workspacePath: string } };
/** The spaces the scan reads and links; new ones come from `createSpaceForMember`. */
function fakeDatabase({ spaceLimit = Number.POSITIVE_INFINITY } = {}) {
const spaces: Array<{ id: string; name: string; workspacePath: string | null }> = [];
createSpaceForMember.mockImplementation(async (_prisma: unknown, input: { name: string }) => {
if (spaces.length >= spaceLimit) throw new SpaceLimitError();
const space = { id: `space-${spaces.length + 1}`, name: input.name, workspacePath: null };
spaces.push(space);
return { id: space.id, name: space.name };
});
const prisma = {
space: {
findMany: async () => spaces.filter((space) => space.workspacePath),
update: async ({ where, data }: SpaceUpdate) => {
const space = spaces.find((candidate) => candidate.id === where.id);
if (space) space.workspacePath = data.workspacePath;
},
},
} as unknown as db.PrismaClient;
return { prisma, spaces };
}
async function scanHomeWithOneFile(): Promise<string> {
const home = await mkdtemp(join(tmpdir(), "evoo-scan-"));
@ -14,6 +45,12 @@ async function scanHomeWithOneFile(): Promise<string> {
return home;
}
async function scanHomeWithRepos(...repos: string[]): Promise<string> {
const home = await scanHomeWithOneFile();
for (const repo of repos) await mkdir(join(home, repo, ".git"), { recursive: true });
return home;
}
async function completed(scanId: string, userId: string) {
return await vi.waitFor(
async () => {
@ -25,13 +62,21 @@ async function completed(scanId: string, userId: string) {
);
}
/** The emulated runtime that the shared test setup pins keeps the scan simulated. */
function allowHostScan() {
vi.stubEnv("AGENT_RUNTIME", "");
vi.stubEnv("SANDBOX_PROVIDER", "");
}
describe("onboarding scan", () => {
afterEach(() => {
vi.unstubAllEnvs();
createSpaceForMember.mockReset();
});
it("hides a scan from every account except the one that started it", async () => {
const { scanId } = await startOnboardingScan({ actor: member }, {});
const { prisma } = fakeDatabase();
const { scanId } = await startOnboardingScan({ actor: member }, prisma);
await expect(getScanStatus(scanId, owner.userId)).rejects.toThrow(/Scan not found/);
await expect(getScanStatus(scanId, member.userId)).resolves.toMatchObject({
status: expect.any(String),
@ -40,25 +85,77 @@ describe("onboarding scan", () => {
});
it("never reads the host without a configured scan home", async () => {
const { scanId } = await startOnboardingScan({ actor: owner }, {});
const { prisma } = fakeDatabase();
const { scanId } = await startOnboardingScan({ actor: owner }, prisma);
const status = await completed(scanId, owner.userId);
expect(status.status).toBe("complete");
expect(status.result).toMatchObject({ localRepos: [], fileSummary: { totalFiles: 0 } });
});
it("lets only the deployment owner discover the configured scan home", async () => {
// The shared test setup pins the emulated runtime, which also keeps the scan simulated.
vi.stubEnv("AGENT_RUNTIME", "");
vi.stubEnv("SANDBOX_PROVIDER", "");
const scanHome = await scanHomeWithOneFile();
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app");
const { prisma } = fakeDatabase();
const asMember = await startOnboardingScan({ actor: member, scanHome }, {});
const asMember = await startOnboardingScan({ actor: member, scanHome }, prisma);
const memberStatus = await completed(asMember.scanId, member.userId);
expect(memberStatus.result).toMatchObject({ fileSummary: { totalFiles: 0 } });
expect(createSpaceForMember).not.toHaveBeenCalled();
const asOwner = await startOnboardingScan({ actor: owner, scanHome }, {});
const asOwner = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const ownerStatus = await completed(asOwner.scanId, owner.userId);
expect(ownerStatus.status).toBe("complete");
expect(ownerStatus.result).toMatchObject({ fileSummary: { totalFiles: 1 } });
});
it("links each repository of the deployment owner to one project of its own", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app", ".config/nvim");
const app = await realpath(join(scanHome, "Projects", "app"));
const { prisma, spaces } = fakeDatabase();
const first = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const firstStatus = await completed(first.scanId, owner.userId);
expect(firstStatus.status).toBe("complete");
expect(firstStatus.result).toMatchObject({
localRepos: [{ name: "app" }],
createdProjects: [{ spaceId: "space-1", name: "app", path: app }],
});
expect(createSpaceForMember).toHaveBeenCalledWith(prisma, {
currentSpaceId: owner.spaceId,
userId: owner.userId,
name: "app",
});
expect(spaces).toEqual([{ id: "space-1", name: "app", workspacePath: app }]);
// Scanning again finds the linked project instead of creating a second one.
const again = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const againStatus = await completed(again.scanId, owner.userId);
expect(againStatus.result).toMatchObject({ createdProjects: [] });
expect(createSpaceForMember).toHaveBeenCalledTimes(1);
});
it("joins a running scan instead of linking its repositories twice", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app");
const { prisma, spaces } = fakeDatabase();
const first = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const second = await startOnboardingScan({ actor: owner, scanHome }, prisma);
expect(second.scanId).toBe(first.scanId);
await completed(first.scanId, owner.userId);
expect(spaces).toHaveLength(1);
});
it("stops creating projects at the space limit and still completes", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/a", "Projects/b", "Projects/c");
const { prisma } = fakeDatabase({ spaceLimit: 1 });
const { scanId } = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const status = await completed(scanId, owner.userId);
expect(status.status).toBe("complete");
expect(status.result).toMatchObject({ createdProjects: [{ spaceId: "space-1" }] });
expect(createSpaceForMember).toHaveBeenCalledTimes(2);
});
});

View File

@ -1,9 +1,16 @@
// Onboarding scan — real filesystem scan, Git analysis, and profile compilation.
// Runs synchronously on the host machine (no sandbox needed).
import { spawn } from "node:child_process";
import { access, readdir, stat } from "node:fs/promises";
import { dirname, join } from "node:path";
import { access, readdir, realpath, stat } from "node:fs/promises";
import { basename, dirname, join } from "node:path";
import type { Actor } from "@rakazo/contracts";
import type { PrismaClient } from "@rakazo/db";
import {
createSpaceForMember,
InvalidSpaceNameError,
SpaceLimitError,
wireProjectWorkspace,
} from "@rakazo/db";
// ---------------------------------------------------------------------------
// Types
@ -54,10 +61,9 @@ interface ProfileSummary {
}
interface CreatedProject {
id: string;
name: string;
spaceId: string;
description: string;
name: string;
path: string;
}
interface ScanRecord {
@ -115,8 +121,10 @@ async function runSimulatedScan(record: ScanRecord) {
record.completedAt = new Date();
}
type ScanActor = Pick<Actor, "userId" | "spaceId" | "isDeploymentOwner">;
export interface OnboardingScanRequest {
actor: Pick<Actor, "userId" | "isDeploymentOwner">;
actor: ScanActor;
botId?: string;
/** Home-style directory to discover. Unset keeps the scan off the host filesystem. */
scanHome?: string;
@ -124,8 +132,12 @@ export interface OnboardingScanRequest {
export async function startOnboardingScan(
{ actor, botId, scanHome }: OnboardingScanRequest,
prisma: any,
prisma: PrismaClient,
): Promise<{ scanId: string }> {
// A repeated start (a remount, a retry) joins the running scan, so no repository is linked twice.
for (const running of scanRegistry.values()) {
if (running.ownerUserId === actor.userId && !running.completedAt) return { scanId: running.id };
}
const scanId = `scan_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
const record: ScanRecord = {
id: scanId,
@ -154,7 +166,7 @@ export async function startOnboardingScan(
await runSimulatedScan(record);
return;
}
const result = await executeFullScan(scanHome, botId ?? "", prisma);
const result = await executeFullScan(scanHome, actor, prisma);
record.status = "complete";
record.progress = 100;
record.message = "Scan complete — profile ready";
@ -189,7 +201,11 @@ export async function getScanStatus(
// Full scan pipeline
// ---------------------------------------------------------------------------
async function executeFullScan(homeDir: string, _botId: string, prisma: any): Promise<ScanResult> {
async function executeFullScan(
homeDir: string,
actor: ScanActor,
prisma: PrismaClient,
): Promise<ScanResult> {
// Phase 1: Local file scan
const [fileSummary, projectFolders] = await scanFiles(homeDir);
@ -200,8 +216,8 @@ async function executeFullScan(homeDir: string, _botId: string, prisma: any): Pr
// Phase 4: Profile compilation from findings
const profileSummary = compileProfile(localRepos, fileSummary, projectFolders);
// Phase 6: Project folder creation
const createdProjects = await createProjectFolders(localRepos, prisma);
// Phase 6: Projects for discovered repositories
const createdProjects = await createProjects(localRepos, actor, prisma);
// Phase 7: Sub-agent role creation
const subAgentRoles = generateSubAgentRoles(profileSummary, localRepos);
@ -413,9 +429,9 @@ async function findGitDirectories(dir: string, depth: number = 0): Promise<strin
if (entry.name === ".git") {
results.push(dirname(fullPath));
} else if (
// Dot folders hold tools and caches (shell and editor plugins), not the person's projects.
!entry.name.startsWith(".") &&
entry.name !== "node_modules" &&
entry.name !== ".git" &&
entry.name !== ".venv" &&
entry.name !== "dist"
) {
results.push(...(await findGitDirectories(fullPath, depth + 1)));
@ -622,38 +638,43 @@ function generateBriefing(
}
// ---------------------------------------------------------------------------
// Phase 5: Project folder creation
// Phase 5: Projects for discovered repositories
// ---------------------------------------------------------------------------
async function createProjectFolders(repos: RepoInfo[], prisma: any): Promise<CreatedProject[]> {
/** Give each discovered repository a project linked to its folder, unless one already links it. */
async function createProjects(
repos: RepoInfo[],
actor: ScanActor,
prisma: PrismaClient,
): Promise<CreatedProject[]> {
const linked = await prisma.space.findMany({
where: { workspacePath: { not: null }, memberships: { some: { userId: actor.userId } } },
select: { workspacePath: true },
});
const linkedFolders = new Set(linked.map((space) => space.workspacePath));
const created: CreatedProject[] = [];
for (const repo of repos) {
// Compare and link the real folder, so the next scan recognizes the projects of this one.
const folder = await realpath(repo.path).catch(() => null);
if (!folder || linkedFolders.has(folder)) continue;
let space: { id: string; name: string };
try {
// Check if space already exists
const existing = await prisma.space.findFirst({ where: { title: repo.name } });
if (existing) {
created.push({
id: existing.id,
name: repo.name,
spaceId: existing.id,
description: repo.remote || repo.path,
});
continue;
}
// Create new space
const space = await prisma.space.create({
data: { title: repo.name, path: repo.path },
space = await createSpaceForMember(prisma, {
currentSpaceId: actor.spaceId,
userId: actor.userId,
name: basename(folder),
});
created.push({
id: space.id,
name: repo.name,
spaceId: space.id,
description: repo.remote || repo.path,
});
} catch {
// Space might already exist or DB unavailable — skip
} catch (error) {
if (error instanceof SpaceLimitError) break;
if (error instanceof InvalidSpaceNameError) continue;
throw error;
}
const workspace = await wireProjectWorkspace(prisma, {
spaceId: space.id,
workspacePath: folder,
});
created.push({ spaceId: space.id, name: space.name, path: workspace.path });
}
return created;

View File

@ -24,6 +24,7 @@ import type {
ConnectorRegistry,
EncryptedSecretStore,
IntegrationProviderSettings,
MediaServices,
MemoryProviderResolver,
PiOAuthLogins,
RemoteConnectorDependencies,
@ -166,6 +167,7 @@ import {
renameMainConversation,
} from "./main-conversation.js";
import { buildMcpUpdateMaterial } from "./mcp-material.js";
import { createMediaRouter } from "./media.js";
import {
disconnectMemoryProvider,
getMemoryProviderConfig,
@ -463,6 +465,8 @@ export interface RouterDeps {
remoteConnectors?: RemoteConnectorDependencies;
artifacts: ArtifactStore;
dataDir: string;
/** Null when no media sidecar (or scripted emulator) is configured; Media Studio then stays hidden. */
mediaServices?: MediaServices | null;
/** Present when the external messaging surface is enabled. */
messaging?: { enabled: boolean; providers: string[]; openSignup: boolean };
env: {
@ -526,6 +530,11 @@ export function createRouter(deps: RouterDeps) {
});
return os.router({
media: createMediaRouter({
prisma: deps.prisma,
services: deps.mediaServices ?? null,
dataDir: deps.dataDir,
}),
aiConsent: {
status: authed.aiConsent.status.handler(({ context, input }) =>
aiConsentStatus(deps, context.actor, input),
@ -569,6 +578,10 @@ export function createRouter(deps: RouterDeps) {
spaceNavigationDto(deps, context.actor, repos, groupRepos),
),
create: authed.spaces.create.handler(async ({ context, input }) => {
// A linked folder lives on the host, and the host is the deployment owner's machine.
if (input.workspacePath && !context.actor.isDeploymentOwner) {
throw new ORPCError("FORBIDDEN");
}
let space: { id: string; name: string };
try {
space = await createSpaceForMember(deps.prisma, {
@ -2230,10 +2243,7 @@ export function createRouter(deps: RouterDeps) {
}
}
if (isProjectWorkspacePath(storedPath)) {
const workspaceRoot = await requireProjectWorkspaceRoot(
deps.prisma,
context.actor.spaceId,
);
const workspaceRoot = await requireProjectWorkspaceRoot(deps.prisma, context.actor);
const entries = await listProjectWorkspaceFiles(workspaceRoot, storedPath);
return entries.map((entry) => ({
...entry,
@ -2281,10 +2291,7 @@ export function createRouter(deps: RouterDeps) {
}
}
if (isProjectWorkspacePath(storedPath)) {
const workspaceRoot = await requireProjectWorkspaceRoot(
deps.prisma,
context.actor.spaceId,
);
const workspaceRoot = await requireProjectWorkspaceRoot(deps.prisma, context.actor);
const bytes = await readProjectWorkspaceFile(workspaceRoot, storedPath, {
maxBytes: MAX_COMPUTER_TEXT_FILE_BYTES,
});
@ -4894,10 +4901,13 @@ function mapUpdaterError(error: unknown): never {
async function requireProjectWorkspaceRoot(
prisma: Pick<PrismaClient, "space">,
spaceId: string,
actor: Actor,
): Promise<string> {
// Only the deployment owner may link a host folder, so only they may browse one. This also
// closes folders that other members linked before linking was limited to the owner.
if (!actor.isDeploymentOwner) throw new ORPCError("FORBIDDEN");
const space = await prisma.space.findUnique({
where: { id: spaceId },
where: { id: actor.spaceId },
select: { workspacePath: true },
});
if (!space?.workspacePath) {
@ -5009,7 +5019,8 @@ async function spaceNavigationDto(
memberships.length > 1 &&
!spacesWithContent.has(membership.spaceId) &&
(membership.space.deletingAt === null || membership.space.deletingAt < staleClaimBefore),
workspace: projectWorkspaceDto(membership.space),
// A linked folder is on the owner's host; other members see neither its path nor remote.
workspace: actor.isDeploymentOwner ? projectWorkspaceDto(membership.space) : null,
bots: spaceBots.map((bot) => ({
id: bot.id,
spaceId: bot.spaceId,

View File

@ -0,0 +1,155 @@
import http from "node:http";
import path from "node:path";
import { _electron as electron, expect, test } from "@playwright/test";
import type { RakazoDesktop, RakazoDesktopBrowserPreviewState } from "@rakazo/contracts";
const fixture = `<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Rakazo desktop browser</title></head>
<body><main>App with a browser panel</main></body>
</html>`;
type Bridge = NonNullable<RakazoDesktop["browserPreview"]>;
test("the in-app browser is a native view the app window lays over its panel", async () => {
const pages = http.createServer((req, res) => {
res.setHeader("content-type", "text/html");
if (req.url === "/second") {
res.end("<!doctype html><title>Second page</title><p>second</p>");
return;
}
res.end(`<!doctype html><title>First page</title>
<a id="next" href="/second">next</a>
<a id="popup" href="/second" target="_blank">popup</a>
<a id="file" href="file:///etc/hosts">file</a>`);
});
await new Promise<void>((resolve) => pages.listen(0, "127.0.0.1", resolve));
const address = pages.address();
if (!address || typeof address === "string") throw new Error("No fixture server");
const origin = `http://127.0.0.1:${address.port}`;
const app = await electron.launch({
args: ["."],
cwd: path.resolve(import.meta.dirname, ".."),
env: {
...process.env,
RAKAZO_WEB_URL: `data:text/html;charset=utf-8,${encodeURIComponent(fixture)}`,
},
});
try {
const page = await app.firstWindow();
await expect(page.getByText("App with a browser panel")).toBeVisible();
const call = <K extends keyof Bridge>(method: K, ...args: unknown[]) =>
page.evaluate(
([name, values]) => {
const bridge = (window as typeof window & { rakazoDesktop?: RakazoDesktop }).rakazoDesktop
?.browserPreview as unknown as Record<
string,
(...input: unknown[]) => Promise<unknown>
>;
return bridge[name as string]!(...(values as unknown[]));
},
[method, args] as const,
);
const state = () => call("state") as Promise<RakazoDesktopBrowserPreviewState>;
const view = () =>
app.evaluate(({ BrowserWindow }) => {
const win = BrowserWindow.getAllWindows()[0]!;
const child = win.contentView.children[0] as
| (Electron.View & { webContents?: Electron.WebContents })
| undefined;
return child
? {
count: win.contentView.children.length,
visible: child.getVisible(),
bounds: child.getBounds(),
url: child.webContents?.getURL(),
partition: child.webContents?.session === win.webContents.session,
sandbox: child.webContents?.getLastWebPreferences()?.sandbox,
nodeIntegration: child.webContents?.getLastWebPreferences()?.nodeIntegration,
}
: null;
});
// Pages load in a native view in their own session, never the app's.
await call("navigate", `${origin}/`);
await expect.poll(async () => (await state()).title).toBe("First page");
expect(await view()).toMatchObject({
count: 1,
visible: false,
url: `${origin}/`,
partition: false,
sandbox: true,
nodeIntegration: false,
});
await call("show", { x: 400, y: 50, width: 300, height: 200 });
expect(await view()).toMatchObject({
visible: true,
bounds: { x: 400, y: 50, width: 300, height: 200 },
});
// Renderer bounds outside the window are clamped to it.
await call("show", { x: -10, y: -10, width: 100_000, height: 100_000 });
const [width, height] = await app.evaluate(({ BrowserWindow }) =>
BrowserWindow.getAllWindows()[0]!.getContentSize(),
);
expect((await view())?.bounds).toEqual({ x: 0, y: 0, width, height });
const contents = () =>
app.evaluate(({ BrowserWindow }, script) => {
const win = BrowserWindow.getAllWindows()[0]!;
const child = win.contentView.children[0] as Electron.View & {
webContents: Electron.WebContents;
};
return child.webContents.executeJavaScript(script, true);
}, "document.getElementById('next').click()");
await contents();
await expect.poll(async () => (await state()).title).toBe("Second page");
expect(await state()).toMatchObject({ canGoBack: true, canGoForward: false, error: null });
await call("back");
await expect.poll(async () => (await state()).title).toBe("First page");
expect(await state()).toMatchObject({ canGoForward: true });
await call("forward");
await expect.poll(async () => (await state()).url).toBe(`${origin}/second`);
// New-window links stay in the one view; file links never load.
await call("back");
await expect.poll(async () => (await state()).title).toBe("First page");
await app.evaluate(({ BrowserWindow }) => {
const child = BrowserWindow.getAllWindows()[0]!.contentView.children[0] as Electron.View & {
webContents: Electron.WebContents;
};
return child.webContents.executeJavaScript("document.getElementById('popup').click()", true);
});
await expect.poll(async () => (await state()).url).toBe(`${origin}/second`);
expect(await app.evaluate(({ BrowserWindow }) => BrowserWindow.getAllWindows().length)).toBe(1);
await call("back");
await expect.poll(async () => (await state()).title).toBe("First page");
await app.evaluate(({ BrowserWindow }) => {
const child = BrowserWindow.getAllWindows()[0]!.contentView.children[0] as Electron.View & {
webContents: Electron.WebContents;
};
return child.webContents.executeJavaScript("document.getElementById('file').click()", true);
});
await page.waitForTimeout(300);
expect((await state()).url).toBe(`${origin}/`);
await expect(call("navigate", "file:///etc/hosts")).rejects.toThrow(/http/);
// A dev server that is not running yet shows as an error, then recovers on reload.
pages.close();
await new Promise((resolve) => setTimeout(resolve, 100));
await call("navigate", `${origin}/gone`);
await expect.poll(async () => (await state()).error).toMatch(/ERR_CONNECTION_REFUSED/);
await call("hide");
expect((await view())?.visible).toBe(false);
} finally {
await app.close();
pages.closeAllConnections();
pages.close();
}
});

View File

@ -42,6 +42,7 @@ test("launches with a narrow preload bridge and an isolated renderer", async ()
});
expect(renderer.bridgeKeys).toEqual([
"browserPreview",
"files",
"localSettings",
"oauth",

View File

@ -0,0 +1,2 @@
.build/
.swiftpm/

View File

@ -0,0 +1,18 @@
// swift-tools-version:5.9
import PackageDescription
// PhotosHelperCore holds everything that does not touch PhotoKit (argument parsing,
// cursors, JSON envelopes, the export path guard, hashing) so XCTest runs without
// Photos access. The executable target is the thin PhotoKit layer.
let package = Package(
name: "photos-helper",
platforms: [.macOS(.v13)],
products: [
.executable(name: "photos-helper", targets: ["photos-helper"])
],
targets: [
.target(name: "PhotosHelperCore"),
.executableTarget(name: "photos-helper", dependencies: ["PhotosHelperCore"]),
.testTarget(name: "PhotosHelperCoreTests", dependencies: ["PhotosHelperCore"]),
]
)

View File

@ -0,0 +1,142 @@
import Foundation
public enum ExportVariant: String, Equatable {
/// The untouched original resource, downloaded from iCloud when needed.
case original
/// A smaller rendition of the current version (720p video, 1280 px JPEG).
case proxy
}
public enum Command: Equatable {
/// `prompt: false` only reads the current status and never shows the system dialog.
case authorize(prompt: Bool)
case list(cursor: ListCursor?, limit: Int, since: Date?)
case asset(localIdentifier: String)
case export(localIdentifier: String, destination: String, variant: ExportVariant)
case changes(token: String)
}
public let usageText = """
usage: photos-helper authorize [--no-prompt]
photos-helper list [--cursor <opaque>] [--limit <1-1000>] [--since <iso8601>]
photos-helper asset <localIdentifier>
photos-helper export <localIdentifier> --dest <dir> [--original|--proxy]
photos-helper changes --token <opaque>
"""
/// Parses argv without the program name. Unknown flags, repeated flags and missing
/// values are usage errors so a typo never silently changes behaviour.
public func parseArguments(_ arguments: [String]) throws -> Command {
guard let name = arguments.first else { throw HelperError(.usage, usageText) }
var options = try OptionReader(Array(arguments.dropFirst()))
switch name {
case "authorize":
let noPrompt = options.flag("--no-prompt")
try options.finish(positionals: 0)
return .authorize(prompt: !noPrompt)
case "list":
let cursor = try options.value("--cursor").map(ListCursor.decode)
let limit = try options.value("--limit").map(parseLimit) ?? Limits.defaultPage
let since = try options.value("--since").map { value -> Date in
guard let date = parseTimestamp(value) else {
throw HelperError(.usage, "--since must be an ISO 8601 timestamp.")
}
return date
}
try options.finish(positionals: 0)
return .list(cursor: cursor, limit: limit, since: since)
case "asset":
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .asset(localIdentifier: identifier)
case "export":
guard let destination = options.value("--dest") else {
throw HelperError(.usage, "export needs --dest <dir>.")
}
let original = options.flag("--original")
let proxy = options.flag("--proxy")
if original && proxy { throw HelperError(.usage, "Choose --original or --proxy, not both.") }
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .export(
localIdentifier: identifier,
destination: destination,
variant: proxy ? .proxy : .original)
case "changes":
guard let token = options.value("--token"), !token.isEmpty else {
throw HelperError(.usage, "changes needs --token <opaque>.")
}
try options.finish(positionals: 0)
return .changes(token: token)
default:
throw HelperError(.usage, usageText)
}
}
func parseLimit(_ value: String) throws -> Int {
guard let limit = Int(value), (1...Limits.maxPage).contains(limit) else {
throw HelperError(.usage, "--limit must be between 1 and \(Limits.maxPage).")
}
return limit
}
func validIdentifier(_ value: String) throws -> String {
guard !value.isEmpty, value.utf16.count <= Limits.externalId, !value.hasPrefix("-"),
!value.unicodeScalars.contains(where: { CharacterSet.controlCharacters.contains($0) })
else { throw HelperError(.usage, "Invalid local identifier.") }
return value
}
private struct OptionReader {
private var values: [String: String] = [:]
private var flags: Set<String> = []
private var consumed: Set<String> = []
private(set) var positionals: [String] = []
private static let valueOptions: Set<String> = ["--cursor", "--limit", "--since", "--dest", "--token"]
private static let flagOptions: Set<String> = ["--no-prompt", "--original", "--proxy"]
init(_ arguments: [String]) throws {
var index = 0
while index < arguments.count {
let argument = arguments[index]
if Self.valueOptions.contains(argument) {
guard index + 1 < arguments.count else {
throw HelperError(.usage, "\(argument) needs a value.")
}
guard values[argument] == nil else { throw HelperError(.usage, "\(argument) given twice.") }
values[argument] = arguments[index + 1]
index += 2
} else if Self.flagOptions.contains(argument) {
guard !flags.contains(argument) else { throw HelperError(.usage, "\(argument) given twice.") }
flags.insert(argument)
index += 1
} else if argument.hasPrefix("--") {
throw HelperError(.usage, "Unknown option \(argument).")
} else {
positionals.append(argument)
index += 1
}
}
}
mutating func value(_ name: String) -> String? {
consumed.insert(name)
return values[name]
}
mutating func flag(_ name: String) -> Bool {
consumed.insert(name)
return flags.contains(name)
}
/// Rejects options that belong to another command and surplus positionals.
func finish(positionals expected: Int) throws {
for name in Set(values.keys).union(flags) where !consumed.contains(name) {
throw HelperError(.usage, "\(name) is not valid here.")
}
guard positionals.count == expected else {
throw HelperError(.usage, expected == 0 ? "Unexpected argument." : "Expected \(expected) argument.")
}
}
}

View File

@ -0,0 +1,37 @@
import Foundation
/// A failure the desktop app maps by `code`. Messages never contain library content
/// beyond the identifier the caller passed in.
public struct HelperError: Error, Equatable {
public enum Code: String, Codable, CaseIterable {
case usage
case notAuthorized = "not_authorized"
case notFound = "not_found"
case pathRejected = "path_rejected"
case destinationExists = "destination_exists"
case network
case exportFailed = "export_failed"
case `internal`
}
public let code: Code
public let message: String
public init(_ code: Code, _ message: String) {
self.code = code
self.message = message
}
/// sysexits(3) style codes so a shell user can tell failure classes apart.
public var exitCode: Int32 {
switch code {
case .usage: return 64
case .notFound: return 66
case .pathRejected, .destinationExists: return 73
case .exportFailed: return 74
case .network: return 75
case .notAuthorized: return 77
case .internal: return 70
}
}
}

View File

@ -0,0 +1,249 @@
import Foundation
/// Mirrors `ClientMediaItemSchema` in packages/contracts/src/media.ts. Every key is
/// always present (nullable fields encode as `null`) because the schema has no optional keys.
public struct ClientMediaItem: Encodable, Equatable {
public enum Kind: String, Encodable {
case video
case photo
case livePhoto = "live_photo"
}
public var externalId: String
public var kind: Kind
public var capturedAt: String?
public var durationMs: Int?
public var width: Int?
public var height: Int?
public var fps: Double?
public var gpsLat: Double?
public var gpsLon: Double?
public var deviceModel: String?
public var albumNames: [String]
public var isFavorite: Bool
public var fileName: String?
public init(
externalId: String,
kind: Kind,
capturedAt: Date?,
durationSeconds: Double?,
width: Int?,
height: Int?,
latitude: Double?,
longitude: Double?,
albumNames: [String],
isFavorite: Bool,
fileName: String?
) {
self.externalId = externalId
self.kind = kind
self.capturedAt = capturedAt.map(formatTimestamp)
if let durationSeconds, durationSeconds.isFinite, durationSeconds > 0 {
durationMs = Int((durationSeconds * 1000).rounded())
} else {
durationMs = nil
}
self.width = width.flatMap { $0 > 0 ? $0 : nil }
self.height = height.flatMap { $0 > 0 ? $0 : nil }
// PhotoKit exposes neither frame rate nor camera model without loading the
// file; the server reads both with ffprobe after upload.
fps = nil
deviceModel = nil
let coordinate = roundedCoordinate(latitude: latitude, longitude: longitude)
gpsLat = coordinate?.latitude
gpsLon = coordinate?.longitude
self.albumNames = Array(
albumNames.map { clip($0, maxCharacters: Limits.albumName) }
.filter { !$0.isEmpty }
.prefix(Limits.albumCount))
self.isFavorite = isFavorite
self.fileName = fileName.map { clip($0, maxCharacters: Limits.fileName) }
}
enum CodingKeys: String, CodingKey {
case externalId, kind, capturedAt, durationMs, width, height, fps, gpsLat, gpsLon
case deviceModel, albumNames, isFavorite, fileName
}
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(externalId, forKey: .externalId)
try container.encode(kind, forKey: .kind)
// `encode` on an Optional writes `null`, unlike the synthesized `encodeIfPresent`.
try container.encode(capturedAt, forKey: .capturedAt)
try container.encode(durationMs, forKey: .durationMs)
try container.encode(width, forKey: .width)
try container.encode(height, forKey: .height)
try container.encode(fps, forKey: .fps)
try container.encode(gpsLat, forKey: .gpsLat)
try container.encode(gpsLon, forKey: .gpsLon)
try container.encode(deviceModel, forKey: .deviceModel)
try container.encode(albumNames, forKey: .albumNames)
try container.encode(isFavorite, forKey: .isFavorite)
try container.encode(fileName, forKey: .fileName)
}
}
/// Bounds from the contract schema, applied here so one odd asset cannot make a whole
/// page fail validation in the app.
public enum Limits {
public static let externalId = 200
public static let albumName = 200
public static let albumCount = 50
public static let fileName = 255
public static let defaultPage = 500
public static let maxPage = 1000
/// Above this many changed assets a full relisting is cheaper than one huge document.
public static let maxChanges = 5000
}
/// Clips to `maxCharacters` UTF-16 units (what zod's `.max()` counts) without splitting
/// a character.
public func clip(_ value: String, maxCharacters: Int) -> String {
if value.utf16.count <= maxCharacters { return value }
var result = ""
var units = 0
for character in value {
units += character.utf16.count
if units > maxCharacters { break }
result.append(character)
}
return result
}
/// Two decimals is about 1 km, matching `roundCoordinate` in @rakazo/media. Precise
/// locations never leave the device.
public func roundedCoordinate(latitude: Double?, longitude: Double?) -> (
latitude: Double, longitude: Double
)? {
guard let latitude, let longitude, latitude.isFinite, longitude.isFinite,
(-90...90).contains(latitude), (-180...180).contains(longitude),
!(latitude == 0 && longitude == 0)
else { return nil }
return ((latitude * 100).rounded() / 100, (longitude * 100).rounded() / 100)
}
/// RFC 3339 in UTC with milliseconds, accepted by zod's `datetime({ offset: true })`.
public func formatTimestamp(_ date: Date) -> String {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
formatter.timeZone = TimeZone(identifier: "UTC")
return formatter.string(from: date)
}
public func parseTimestamp(_ value: String) -> Date? {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
if let date = formatter.date(from: value) { return date }
formatter.formatOptions = [.withInternetDateTime]
return formatter.date(from: value)
}
public enum AuthorizationState: String, Encodable {
case authorized, limited, denied, restricted, notDetermined
}
public struct AuthorizationOutput: Encodable, Equatable {
public let status: AuthorizationState
public init(status: AuthorizationState) { self.status = status }
}
public struct ListOutput: Encodable {
public let items: [ClientMediaItem]
public let nextCursor: String?
/// Library change token captured before enumeration; `changes --token` resumes from it.
public let changeToken: String?
public init(items: [ClientMediaItem], nextCursor: String?, changeToken: String?) {
self.items = items
self.nextCursor = nextCursor
self.changeToken = changeToken
}
enum CodingKeys: String, CodingKey { case items, nextCursor, changeToken }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(items, forKey: .items)
try container.encode(nextCursor, forKey: .nextCursor)
try container.encode(changeToken, forKey: .changeToken)
}
}
public struct AssetResourceInfo: Encodable, Equatable {
public let type: String
public let fileName: String
public let uti: String
public init(type: String, fileName: String, uti: String) {
self.type = type
self.fileName = fileName
self.uti = uti
}
}
public struct AssetOutput: Encodable {
public let item: ClientMediaItem
public let modifiedAt: String?
public let hidden: Bool
public let resources: [AssetResourceInfo]
public init(item: ClientMediaItem, modifiedAt: Date?, hidden: Bool, resources: [AssetResourceInfo]) {
self.item = item
self.modifiedAt = modifiedAt.map(formatTimestamp)
self.hidden = hidden
self.resources = resources
}
enum CodingKeys: String, CodingKey { case item, modifiedAt, hidden, resources }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(item, forKey: .item)
try container.encode(modifiedAt, forKey: .modifiedAt)
try container.encode(hidden, forKey: .hidden)
try container.encode(resources, forKey: .resources)
}
}
public struct ChangesOutput: Encodable {
/// Token to store for the next call; null only when a full relisting is required.
public let token: String?
public let changed: [ClientMediaItem]
public let deleted: [String]
public let fullResyncRequired: Bool
public init(token: String?, changed: [ClientMediaItem], deleted: [String], fullResyncRequired: Bool) {
self.token = token
self.changed = changed
self.deleted = deleted
self.fullResyncRequired = fullResyncRequired
}
public static func fullResync() -> ChangesOutput {
ChangesOutput(token: nil, changed: [], deleted: [], fullResyncRequired: true)
}
enum CodingKeys: String, CodingKey { case token, changed, deleted, fullResyncRequired }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(token, forKey: .token)
try container.encode(changed, forKey: .changed)
try container.encode(deleted, forKey: .deleted)
try container.encode(fullResyncRequired, forKey: .fullResyncRequired)
}
}
public struct ExportOutput: Encodable, Equatable {
public let path: String
public let fileName: String
public let sizeBytes: Int64
public let sha256: String
public init(path: String, fileName: String, sizeBytes: Int64, sha256: String) {
self.path = path
self.fileName = fileName
self.sizeBytes = sizeBytes
self.sha256 = sha256
}
}

View File

@ -0,0 +1,56 @@
import CryptoKit
import Foundation
/// stdout carries exactly one JSON document per successful command; stderr carries
/// JSON lines (`{"progress":…}` while exporting, `{"error":…}` on failure).
public func encodeJSON<T: Encodable>(_ value: T) throws -> String {
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes]
encoder.nonConformingFloatEncodingStrategy = .throw
let data = try encoder.encode(value)
guard let text = String(data: data, encoding: .utf8) else {
throw HelperError(.internal, "Output was not UTF-8.")
}
return text
}
private struct ErrorEnvelope: Encodable {
struct Body: Encodable {
let code: String
let message: String
}
let error: Body
}
public func errorLine(_ error: HelperError) -> String {
let envelope = ErrorEnvelope(error: .init(code: error.code.rawValue, message: error.message))
return (try? encodeJSON(envelope))
?? #"{"error":{"code":"internal","message":"Could not encode the error."}}"#
}
private struct ProgressLine: Encodable {
let progress: Double
}
/// Progress clamped to 0…1 and rounded to three decimals so the line stays short.
public func progressLine(_ fraction: Double) -> String {
let value = fraction.isFinite ? min(1, max(0, fraction)) : 0
return (try? encodeJSON(ProgressLine(progress: (value * 1000).rounded() / 1000)))
?? #"{"progress":0}"#
}
/// Streams the file through SHA-256 in 1 MiB reads so large videos never load fully.
public func sha256Hex(of url: URL) throws -> (hex: String, sizeBytes: Int64) {
let handle = try FileHandle(forReadingFrom: url)
defer { try? handle.close() }
var hasher = SHA256()
var size: Int64 = 0
while true {
let chunk = try handle.read(upToCount: 1 << 20) ?? Data()
if chunk.isEmpty { break }
hasher.update(data: chunk)
size += Int64(chunk.count)
}
let hex = hasher.finalize().map { String(format: "%02x", $0) }.joined()
return (hex, size)
}

View File

@ -0,0 +1,131 @@
import Foundation
/// Listing order: undated assets first, then creation date ascending, ties broken by
/// local identifier. PhotoKit can only sort by date, so ties are ordered here.
public struct SortKey: Comparable, Hashable {
/// Milliseconds since 1970; nil for assets without a creation date.
public let creationMillis: Int64?
public let localIdentifier: String
public init(creationMillis: Int64?, localIdentifier: String) {
self.creationMillis = creationMillis
self.localIdentifier = localIdentifier
}
public init(creationDate: Date?, localIdentifier: String) {
self.init(
creationMillis: creationDate.map { Int64(($0.timeIntervalSince1970 * 1000).rounded(.down)) },
localIdentifier: localIdentifier)
}
public static func < (lhs: SortKey, rhs: SortKey) -> Bool {
if lhs.creationMillis != rhs.creationMillis {
switch (lhs.creationMillis, rhs.creationMillis) {
case (nil, _): return true
case (_, nil): return false
case let (left?, right?): return left < right
}
}
return lhs.localIdentifier < rhs.localIdentifier
}
}
/// Opaque to the app: base64url of `{"v":1,"d":<millis|null>,"id":"<localIdentifier>"}`.
/// The cursor names the last item returned, so listing resumes after it even when
/// assets were added or removed in between.
public struct ListCursor: Equatable {
public let key: SortKey
public init(key: SortKey) { self.key = key }
private struct Payload: Codable {
let v: Int
let d: Int64?
let id: String
enum CodingKeys: String, CodingKey { case v, d, id }
func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(v, forKey: .v)
try container.encode(d, forKey: .d)
try container.encode(id, forKey: .id)
}
}
public func encoded() -> String {
let payload = Payload(v: 1, d: key.creationMillis, id: key.localIdentifier)
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys]
// Encoding a struct of plain values cannot fail.
let data = (try? encoder.encode(payload)) ?? Data()
return base64URLEncode(data)
}
public static func decode(_ value: String) throws -> ListCursor {
guard value.count <= 2048, let data = base64URLDecode(value),
let payload = try? JSONDecoder().decode(Payload.self, from: data),
payload.v == 1, !payload.id.isEmpty
else { throw HelperError(.usage, "Invalid cursor.") }
return ListCursor(key: SortKey(creationMillis: payload.d, localIdentifier: payload.id))
}
}
public func base64URLEncode(_ data: Data) -> String {
data.base64EncodedString()
.replacingOccurrences(of: "+", with: "-")
.replacingOccurrences(of: "/", with: "_")
.replacingOccurrences(of: "=", with: "")
}
public func base64URLDecode(_ value: String) -> Data? {
guard value.allSatisfy({ $0.isASCII && ($0.isLetter || $0.isNumber || $0 == "-" || $0 == "_") })
else { return nil }
var base64 = value.replacingOccurrences(of: "-", with: "+").replacingOccurrences(of: "_", with: "/")
while base64.count % 4 != 0 { base64.append("=") }
return Data(base64Encoded: base64)
}
public struct Page {
/// Positions in the source sequence, in listing order.
public let indices: [Int]
public let hasMore: Bool
}
/// Selects the next page from a sequence sorted by creation date only (ties in any
/// order). Binary search finds the first candidate date; a whole same-date group is
/// collected before sorting so tie order never depends on PhotoKit.
public func nextPage(count: Int, key: (Int) -> SortKey, after cursor: ListCursor?, limit: Int)
-> Page
{
precondition(limit > 0)
var start = 0
if let cursor, let cursorMillis = cursor.key.creationMillis {
var low = 0
var high = count
while low < high {
let mid = (low + high) / 2
let millis = key(mid).creationMillis
if millis == nil || millis! < cursorMillis { low = mid + 1 } else { high = mid }
}
start = low
}
var candidates: [(index: Int, key: SortKey)] = []
var stoppedEarly = false
var index = start
while index < count {
let current = key(index)
if candidates.count >= limit, let last = candidates.last, current.creationMillis != last.key.creationMillis {
// Dates ascend, so everything from here on sorts after the collected group.
stoppedEarly = true
break
}
if cursor == nil || cursor!.key < current {
candidates.append((index, current))
}
index += 1
}
candidates.sort { $0.key < $1.key }
let hasMore = stoppedEarly || candidates.count > limit
return Page(indices: candidates.prefix(limit).map(\.index), hasMore: hasMore)
}

View File

@ -0,0 +1,72 @@
import Foundation
/// Where an export is written: a final path and a sibling temporary path inside the
/// destination directory.
public struct ExportDestination: Equatable {
public let directory: URL
public let finalURL: URL
public let temporaryURL: URL
public let fileName: String
}
/// Resolves the export target. The directory must be absolute, free of `..`, existing
/// and a directory after resolving symlinks (macOS temp dirs live behind /var). The file
/// name comes from Photos, so it is reduced to one safe component and the joined path is
/// checked to stay a direct child of the resolved directory. Existing files are never
/// overwritten.
public func resolveExportDestination(
directory: String,
fileName rawFileName: String,
temporarySuffix: String = UUID().uuidString,
fileManager: FileManager = .default
) throws -> ExportDestination {
guard directory.hasPrefix("/"), !directory.contains("\0") else {
throw HelperError(.pathRejected, "--dest must be an absolute directory path.")
}
let components = directory.split(separator: "/", omittingEmptySubsequences: true)
guard !components.contains(where: { $0 == ".." || $0 == "." }) else {
throw HelperError(.pathRejected, "--dest must not contain . or .. segments.")
}
let resolved = URL(fileURLWithPath: directory, isDirectory: true).resolvingSymlinksInPath()
.standardizedFileURL
var isDirectory: ObjCBool = false
guard fileManager.fileExists(atPath: resolved.path, isDirectory: &isDirectory), isDirectory.boolValue
else { throw HelperError(.pathRejected, "--dest must be an existing directory.") }
let fileName = sanitizeFileName(rawFileName)
let finalURL = resolved.appendingPathComponent(fileName, isDirectory: false).standardizedFileURL
let temporaryURL = resolved.appendingPathComponent(
".\(fileName).\(temporarySuffix).partial", isDirectory: false
).standardizedFileURL
for candidate in [finalURL, temporaryURL]
where candidate.deletingLastPathComponent().path != resolved.path {
throw HelperError(.pathRejected, "Export path left the destination directory.")
}
// lstat semantics: a dangling symlink at the final path also counts as existing.
if (try? fileManager.attributesOfItem(atPath: finalURL.path)) != nil {
throw HelperError(.destinationExists, "The destination file already exists.")
}
return ExportDestination(
directory: resolved, finalURL: finalURL, temporaryURL: temporaryURL, fileName: fileName)
}
/// One path component with no separators, control characters or leading dots, at most
/// 200 UTF-8 bytes with the extension kept.
public func sanitizeFileName(_ raw: String) -> String {
var cleaned = String(
raw.unicodeScalars.map { scalar -> Character in
if scalar == "/" || scalar == "\\" || scalar == ":" { return "_" }
if CharacterSet.controlCharacters.contains(scalar) { return "_" }
return Character(scalar)
})
while cleaned.hasPrefix(".") || cleaned.hasPrefix(" ") { cleaned.removeFirst() }
cleaned = cleaned.trimmingCharacters(in: .whitespaces)
if cleaned.isEmpty { return "asset" }
let maxBytes = 200
if cleaned.utf8.count <= maxBytes { return cleaned }
let ext = (cleaned as NSString).pathExtension
let suffix = ext.isEmpty || ext.utf8.count > 16 ? "" : ".\(ext)"
var stem = suffix.isEmpty ? cleaned : String(cleaned.dropLast(suffix.count))
while stem.utf8.count + suffix.utf8.count > maxBytes { stem.removeLast() }
return stem + suffix
}

View File

@ -0,0 +1,148 @@
import AVFoundation
import AppKit
import Foundation
import Photos
import PhotosHelperCore
/// Writes one asset into the destination directory: first to a hidden temporary file,
/// then renamed into place after hashing, so a cancelled export never leaves a file
/// that looks complete.
struct Exporter {
func export(
localIdentifier: String,
directory: String,
variant: ExportVariant,
progress: @escaping (Double) -> Void
) async throws -> ExportOutput {
guard
let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil)
.firstObject
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
guard let resource = originalResource(for: asset, variant: variant) else {
throw HelperError(.notFound, "The asset has no exportable resource.")
}
var fileName = resource.originalFilename
if variant == .proxy {
let stem = (fileName as NSString).deletingPathExtension
fileName = stem + (asset.mediaType == .video ? ".mp4" : ".jpg")
}
let destination = try resolveExportDestination(directory: directory, fileName: fileName)
let fileManager = FileManager.default
do {
switch (variant, asset.mediaType) {
case (.original, _):
try await writeResource(resource, to: destination.temporaryURL, progress: progress)
case (.proxy, .video):
try await writeVideoProxy(asset, to: destination.temporaryURL, progress: progress)
case (.proxy, _):
try await writeImageProxy(asset, to: destination.temporaryURL, progress: progress)
}
let digest = try sha256Hex(of: destination.temporaryURL)
// moveItem refuses an existing destination, so nothing is ever overwritten.
try fileManager.moveItem(at: destination.temporaryURL, to: destination.finalURL)
progress(1)
return ExportOutput(
path: destination.finalURL.path, fileName: destination.fileName,
sizeBytes: digest.sizeBytes, sha256: digest.hex)
} catch {
try? fileManager.removeItem(at: destination.temporaryURL)
if let helperError = error as? HelperError { throw helperError }
throw mapExportError(error)
}
}
/// Original bytes through PHAssetResourceManager; downloads from iCloud when the
/// library keeps only optimized copies on this Mac.
private func writeResource(
_ resource: PHAssetResource, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHAssetResourceRequestOptions()
options.isNetworkAccessAllowed = true
options.progressHandler = { value in progress(value) }
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
PHAssetResourceManager.default().writeData(for: resource, toFile: url, options: options) {
error in
if let error { continuation.resume(throwing: error) } else { continuation.resume() }
}
}
}
/// 720p H.264 MP4 of the current version, without source metadata.
private func writeVideoProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHVideoRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.version = .current
options.progressHandler = { value, _, _, _ in progress(value * 0.5) }
let session: AVAssetExportSession = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestExportSession(
forVideo: asset, options: options, exportPreset: AVAssetExportPreset1280x720
) { session, info in
if let session {
continuation.resume(returning: session)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not prepare the video export."))
}
}
}
session.outputURL = url
session.outputFileType = .mp4
session.shouldOptimizeForNetworkUse = true
session.metadata = []
session.metadataItemFilter = .forSharing()
await session.export()
guard session.status == .completed else {
throw session.error ?? HelperError(.exportFailed, "The video export did not finish.")
}
}
/// JPEG of the current version, at most 1280 px on the long edge, without EXIF/GPS.
private func writeImageProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHImageRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.resizeMode = .exact
options.version = .current
options.isSynchronous = false
options.progressHandler = { value, _, _, _ in progress(value) }
// highQualityFormat calls the handler exactly once.
let image: NSImage = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestImage(
for: asset, targetSize: CGSize(width: 1280, height: 1280), contentMode: .aspectFit,
options: options
) { image, info in
if let image {
continuation.resume(returning: image)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not render the photo."))
}
}
}
guard let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil),
let data = NSBitmapImageRep(cgImage: cgImage).representation(
using: .jpeg, properties: [.compressionFactor: 0.85])
else { throw HelperError(.exportFailed, "Could not encode the photo.") }
try data.write(to: url, options: .withoutOverwriting)
}
}
/// iCloud and network failures are retryable; everything else is an export failure.
func mapExportError(_ error: Error) -> HelperError {
if let photosError = error as? PHPhotosError,
photosError.code == .networkAccessRequired || photosError.code == .networkError
{
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
if (error as NSError).domain == NSURLErrorDomain {
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
return HelperError(.exportFailed, "The export failed.")
}

View File

@ -0,0 +1,183 @@
import Foundation
import Photos
import PhotosHelperCore
/// Read-only PhotoKit queries. Nothing here writes to the library.
struct Library {
private let library = PHPhotoLibrary.shared()
/// Images and videos only; audio and unknown media types are not part of Media Studio.
/// PhotoKit sorts by date only; the store places missing dates first, which is what
/// `SortKey` assumes, and `nextPage` orders ties itself.
private func fetchOptions(since: Date?) -> PHFetchOptions {
let options = PHFetchOptions()
options.includeHiddenAssets = false
var predicates = [
NSPredicate(
format: "mediaType == %ld || mediaType == %ld",
PHAssetMediaType.image.rawValue, PHAssetMediaType.video.rawValue)
]
if let since {
predicates.append(NSPredicate(format: "modificationDate >= %@", since as NSDate))
}
options.predicate = NSCompoundPredicate(andPredicateWithSubpredicates: predicates)
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: true)]
return options
}
func list(after cursor: ListCursor?, limit: Int, since: Date?) throws -> ListOutput {
// Captured before enumeration so changes made while listing show up next time.
let changeToken = try encodeToken(library.currentChangeToken)
let result = PHAsset.fetchAssets(with: fetchOptions(since: since))
let page = nextPage(
count: result.count,
key: { index in
let asset = result.object(at: index)
return SortKey(creationDate: asset.creationDate, localIdentifier: asset.localIdentifier)
},
after: cursor,
limit: limit)
let assets = page.indices.map { result.object(at: $0) }
let items = assets.compactMap(makeItem(for:))
let nextCursor = page.hasMore
? assets.last.map {
ListCursor(key: SortKey(creationDate: $0.creationDate, localIdentifier: $0.localIdentifier))
.encoded()
}
: nil
return ListOutput(items: items, nextCursor: nextCursor, changeToken: changeToken)
}
func asset(_ localIdentifier: String) throws -> AssetOutput {
guard let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil).firstObject,
let item = makeItem(for: asset)
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
let resources = PHAssetResource.assetResources(for: asset).map {
AssetResourceInfo(
type: resourceTypeName($0.type), fileName: $0.originalFilename, uti: $0.uniformTypeIdentifier)
}
return AssetOutput(
item: item, modifiedAt: asset.modificationDate, hidden: asset.isHidden, resources: resources)
}
/// Persistent change history (macOS 13). An expired token or missing history asks the
/// caller for a full relisting instead of guessing.
func changes(since encodedToken: String) throws -> ChangesOutput {
guard let token = decodeToken(encodedToken) else { return .fullResync() }
let current = try encodeToken(library.currentChangeToken)
var changed = Set<String>()
var deleted = Set<String>()
do {
let result = try library.fetchPersistentChanges(since: token)
for change in result {
let details = try change.changeDetails(for: PHObjectType.asset)
changed.formUnion(details.insertedLocalIdentifiers)
changed.formUnion(details.updatedLocalIdentifiers)
deleted.formUnion(details.deletedLocalIdentifiers)
if changed.count + deleted.count > Limits.maxChanges { break }
}
} catch let error as PHPhotosError
where error.code == .persistentChangeTokenExpired
|| error.code == .persistentChangeDetailsUnavailable
{
return .fullResync()
} catch {
throw HelperError(.internal, "Could not read the Photos change history.")
}
if changed.count + deleted.count > Limits.maxChanges { return .fullResync() }
changed.subtract(deleted)
var items: [ClientMediaItem] = []
var found = Set<String>()
if !changed.isEmpty {
PHAsset.fetchAssets(withLocalIdentifiers: Array(changed), options: nil).enumerateObjects {
asset, _, _ in
found.insert(asset.localIdentifier)
if let item = makeItem(for: asset) { items.append(item) }
}
}
// Assets that vanished between the history read and the fetch count as deleted.
deleted.formUnion(changed.subtracting(found))
items.sort { $0.externalId < $1.externalId }
return ChangesOutput(
token: current, changed: items, deleted: deleted.sorted(), fullResyncRequired: false)
}
private func encodeToken(_ token: PHPersistentChangeToken) throws -> String {
guard
let data = try? NSKeyedArchiver.archivedData(withRootObject: token, requiringSecureCoding: true)
else { throw HelperError(.internal, "Could not encode the Photos change token.") }
return base64URLEncode(data)
}
private func decodeToken(_ value: String) -> PHPersistentChangeToken? {
guard let data = base64URLDecode(value) else { return nil }
return try? NSKeyedUnarchiver.unarchivedObject(ofClass: PHPersistentChangeToken.self, from: data)
}
}
/// Builds the contract item. Returns nil for media types Media Studio does not handle
/// and for identifiers longer than the contract allows.
func makeItem(for asset: PHAsset) -> ClientMediaItem? {
let kind: ClientMediaItem.Kind
switch asset.mediaType {
case .video: kind = .video
case .image: kind = asset.mediaSubtypes.contains(.photoLive) ? .livePhoto : .photo
default: return nil
}
guard asset.localIdentifier.utf16.count <= Limits.externalId else { return nil }
var albums: [String] = []
PHAssetCollection.fetchAssetCollectionsContaining(asset, with: .album, options: nil)
.enumerateObjects { collection, _, stop in
if let title = collection.localizedTitle { albums.append(title) }
if albums.count >= Limits.albumCount { stop.pointee = true }
}
let coordinate = asset.location?.coordinate
return ClientMediaItem(
externalId: asset.localIdentifier,
kind: kind,
capturedAt: asset.creationDate,
durationSeconds: kind == .video ? asset.duration : nil,
width: asset.pixelWidth,
height: asset.pixelHeight,
latitude: coordinate?.latitude,
longitude: coordinate?.longitude,
albumNames: albums,
isFavorite: asset.isFavorite,
fileName: originalResource(for: asset, variant: .original)?.originalFilename)
}
/// The resource holding the untouched original: the video track for videos, the still
/// for photos and Live Photos (the paired motion clip is not exported).
func originalResource(for asset: PHAsset, variant: ExportVariant) -> PHAssetResource? {
let resources = PHAssetResource.assetResources(for: asset)
let preferred: [PHAssetResourceType]
switch (asset.mediaType, variant) {
case (.video, .original): preferred = [.video, .fullSizeVideo]
case (.video, .proxy): preferred = [.fullSizeVideo, .video]
case (_, .original): preferred = [.photo, .fullSizePhoto]
case (_, .proxy): preferred = [.fullSizePhoto, .photo]
}
for type in preferred {
if let match = resources.first(where: { $0.type == type }) { return match }
}
return nil
}
func resourceTypeName(_ type: PHAssetResourceType) -> String {
switch type {
case .photo: return "photo"
case .video: return "video"
case .audio: return "audio"
case .alternatePhoto: return "alternate_photo"
case .fullSizePhoto: return "full_size_photo"
case .fullSizeVideo: return "full_size_video"
case .adjustmentData: return "adjustment_data"
case .adjustmentBasePhoto: return "adjustment_base_photo"
case .pairedVideo: return "paired_video"
case .fullSizePairedVideo: return "full_size_paired_video"
case .adjustmentBasePairedVideo: return "adjustment_base_paired_video"
case .adjustmentBaseVideo: return "adjustment_base_video"
@unknown default: return "other"
}
}

View File

@ -0,0 +1,103 @@
import Foundation
import Photos
import PhotosHelperCore
/// Entry point. Exactly one JSON document goes to stdout on success; failures write
/// `{"error":{code,message}}` to stderr and exit non-zero (see HelperError.exitCode).
@main
struct PhotosHelper {
static func main() async {
do {
let command = try parseArguments(Array(CommandLine.arguments.dropFirst()))
let output = try await run(command)
FileHandle.standardOutput.write(Data((output + "\n").utf8))
exit(0)
} catch let error as HelperError {
fail(error)
} catch {
fail(HelperError(.internal, "Unexpected failure."))
}
}
static func fail(_ error: HelperError) -> Never {
FileHandle.standardError.write(Data((errorLine(error) + "\n").utf8))
exit(error.exitCode)
}
static func run(_ command: Command) async throws -> String {
switch command {
case .authorize(let prompt):
let status = await authorize(prompt: prompt)
return try encodeJSON(AuthorizationOutput(status: status))
case .list(let cursor, let limit, let since):
try requireAccess()
return try encodeJSON(try Library().list(after: cursor, limit: limit, since: since))
case .asset(let localIdentifier):
try requireAccess()
return try encodeJSON(try Library().asset(localIdentifier))
case .export(let localIdentifier, let destination, let variant):
try requireAccess()
let progress = ProgressReporter()
// Local copies report no progress, so a heartbeat repeats the last value; the app
// treats a silent helper as stalled.
let heartbeat = Task {
while !Task.isCancelled {
try? await Task.sleep(nanoseconds: 15_000_000_000)
if !Task.isCancelled { progress.emit(nil) }
}
}
defer { heartbeat.cancel() }
progress.emit(0)
let result = try await Exporter().export(
localIdentifier: localIdentifier, directory: destination, variant: variant,
progress: { fraction in progress.emit(fraction) })
return try encodeJSON(result)
case .changes(let token):
try requireAccess()
return try encodeJSON(try Library().changes(since: token))
}
}
/// `.readWrite` is the lowest PhotoKit level that can read assets: `.addOnly` cannot
/// enumerate or export anything. The helper never writes to the library.
static func authorize(prompt: Bool) async -> AuthorizationState {
var status = PHPhotoLibrary.authorizationStatus(for: .readWrite)
if prompt && status == .notDetermined {
status = await PHPhotoLibrary.requestAuthorization(for: .readWrite)
}
return state(status)
}
static func state(_ status: PHAuthorizationStatus) -> AuthorizationState {
switch status {
case .authorized: return .authorized
case .limited: return .limited
case .denied: return .denied
case .restricted: return .restricted
case .notDetermined: return .notDetermined
@unknown default: return .denied
}
}
/// Serializes progress lines written from PhotoKit callback queues and the heartbeat.
final class ProgressReporter: @unchecked Sendable {
private let lock = NSLock()
private var last = 0.0
/// nil repeats the last reported value.
func emit(_ fraction: Double?) {
lock.lock()
defer { lock.unlock() }
if let fraction { last = max(last, fraction) }
FileHandle.standardError.write(Data((progressLine(last) + "\n").utf8))
}
}
/// Every command except `authorize` runs only with granted access and never prompts.
static func requireAccess() throws {
switch state(PHPhotoLibrary.authorizationStatus(for: .readWrite)) {
case .authorized, .limited: return
default: throw HelperError(.notAuthorized, "Photos access has not been granted.")
}
}
}

View File

@ -0,0 +1,73 @@
import XCTest
@testable import PhotosHelperCore
final class ArgumentsTests: XCTestCase {
func testAuthorizePromptsUnlessAskedNotTo() throws {
XCTAssertEqual(try parseArguments(["authorize"]), .authorize(prompt: true))
XCTAssertEqual(try parseArguments(["authorize", "--no-prompt"]), .authorize(prompt: false))
}
func testListDefaultsAndOptions() throws {
XCTAssertEqual(try parseArguments(["list"]), .list(cursor: nil, limit: 500, since: nil))
let cursor = ListCursor(key: SortKey(creationMillis: 1_700_000_000_000, localIdentifier: "A/L0/001"))
let parsed = try parseArguments([
"list", "--cursor", cursor.encoded(), "--limit", "25", "--since", "2024-01-02T03:04:05Z",
])
XCTAssertEqual(
parsed,
.list(cursor: cursor, limit: 25, since: Date(timeIntervalSince1970: 1_704_164_645)))
}
func testListRejectsBadValues() {
for arguments in [
["list", "--limit", "0"],
["list", "--limit", "1001"],
["list", "--limit", "ten"],
["list", "--limit"],
["list", "--since", "yesterday"],
["list", "--cursor", "not a cursor"],
["list", "--limit", "5", "--limit", "6"],
["list", "--dest", "/tmp"],
["list", "extra"],
["list", "--unknown"],
] {
XCTAssertThrowsError(try parseArguments(arguments), "\(arguments)") { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
}
}
}
func testAssetNeedsExactlyOneIdentifier() throws {
XCTAssertEqual(try parseArguments(["asset", "ABC/L0/001"]), .asset(localIdentifier: "ABC/L0/001"))
XCTAssertThrowsError(try parseArguments(["asset"]))
XCTAssertThrowsError(try parseArguments(["asset", "a", "b"]))
XCTAssertThrowsError(try parseArguments(["asset", String(repeating: "x", count: 201)]))
XCTAssertThrowsError(try parseArguments(["asset", "bad\nid"]))
}
func testExportVariants() throws {
XCTAssertEqual(
try parseArguments(["export", "ID", "--dest", "/tmp/out"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .original))
XCTAssertEqual(
try parseArguments(["export", "--dest", "/tmp/out", "ID", "--proxy"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .proxy))
XCTAssertThrowsError(try parseArguments(["export", "ID"]))
XCTAssertThrowsError(try parseArguments(["export", "ID", "--dest", "/tmp", "--original", "--proxy"]))
}
func testChangesNeedsToken() throws {
XCTAssertEqual(try parseArguments(["changes", "--token", "abc"]), .changes(token: "abc"))
XCTAssertThrowsError(try parseArguments(["changes"]))
XCTAssertThrowsError(try parseArguments(["changes", "--token", ""]))
}
func testUnknownCommandIsUsageError() {
XCTAssertThrowsError(try parseArguments([])) { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
XCTAssertEqual((error as? HelperError)?.exitCode, 64)
}
XCTAssertThrowsError(try parseArguments(["delete", "ID"]))
}
}

View File

@ -0,0 +1,92 @@
import XCTest
@testable import PhotosHelperCore
final class OutputTests: XCTestCase {
private func object(_ json: String) throws -> [String: Any] {
try XCTUnwrap(JSONSerialization.jsonObject(with: Data(json.utf8)) as? [String: Any])
}
func testItemEncodesEveryContractKeyWithExplicitNulls() throws {
let item = ClientMediaItem(
externalId: "ID/L0/001", kind: .photo, capturedAt: nil, durationSeconds: nil, width: 0,
height: nil, latitude: nil, longitude: nil, albumNames: [], isFavorite: false, fileName: nil)
let json = try object(encodeJSON(item))
XCTAssertEqual(
Set(json.keys),
[
"externalId", "kind", "capturedAt", "durationMs", "width", "height", "fps", "gpsLat",
"gpsLon", "deviceModel", "albumNames", "isFavorite", "fileName",
])
for key in ["capturedAt", "durationMs", "width", "height", "fps", "gpsLat", "gpsLon", "deviceModel", "fileName"] {
XCTAssertTrue(json[key] is NSNull, key)
}
}
func testItemNormalizesValues() throws {
let item = ClientMediaItem(
externalId: "V/L0/001", kind: .livePhoto, capturedAt: Date(timeIntervalSince1970: 1_704_164_645.5),
durationSeconds: 12.3456, width: 1920, height: 1080, latitude: 52.520_08, longitude: 13.404_95,
albumNames: ["Trip", "", String(repeating: "a", count: 300)] + Array(repeating: "x", count: 60),
isFavorite: true, fileName: "IMG_0001.HEIC")
let json = try object(encodeJSON(item))
XCTAssertEqual(json["kind"] as? String, "live_photo")
XCTAssertEqual(json["capturedAt"] as? String, "2024-01-02T03:04:05.500Z")
XCTAssertEqual(json["durationMs"] as? Int, 12_346)
XCTAssertEqual(json["gpsLat"] as? Double, 52.52)
XCTAssertEqual(json["gpsLon"] as? Double, 13.4)
let albums = try XCTUnwrap(json["albumNames"] as? [String])
XCTAssertEqual(albums.count, 50)
XCTAssertEqual(albums[1].count, 200)
}
func testInvalidCoordinatesAreDropped() {
XCTAssertNil(roundedCoordinate(latitude: 91, longitude: 0.5))
XCTAssertNil(roundedCoordinate(latitude: .nan, longitude: 1))
XCTAssertNil(roundedCoordinate(latitude: 0, longitude: 0))
XCTAssertNil(roundedCoordinate(latitude: 1, longitude: nil))
}
func testClipCountsUTF16Units() {
XCTAssertEqual(clip("abc", maxCharacters: 5), "abc")
// Each emoji is two UTF-16 units; a half character is never kept.
XCTAssertEqual(clip("😀😀😀", maxCharacters: 5), "😀😀")
}
func testErrorAndProgressLinesAreSingleLineJSON() throws {
let line = errorLine(HelperError(.notFound, "No \"asset\"\nhere."))
XCTAssertFalse(line.contains("\n"))
let error = try XCTUnwrap(try object(line)["error"] as? [String: String])
XCTAssertEqual(error["code"], "not_found")
XCTAssertEqual(error["message"], "No \"asset\"\nhere.")
for (input, expected) in [(0.42424, 0.424), (7, 1), (-1, 0), (.nan, 0)] as [(Double, Double)] {
let progress = progressLine(input)
XCTAssertFalse(progress.contains("\n"))
XCTAssertEqual(try object(progress)["progress"] as? Double ?? -1, expected, accuracy: 1e-9)
}
}
func testListAndChangesKeepNullKeys() throws {
let list = try object(encodeJSON(ListOutput(items: [], nextCursor: nil, changeToken: nil)))
XCTAssertTrue(list["nextCursor"] is NSNull)
XCTAssertTrue(list["changeToken"] is NSNull)
let changes = try object(encodeJSON(ChangesOutput.fullResync()))
XCTAssertTrue(changes["token"] is NSNull)
XCTAssertEqual(changes["fullResyncRequired"] as? Bool, true)
}
func testExitCodesAreDistinctPerFailureClass() {
let codes = HelperError.Code.allCases.map { HelperError($0, "").exitCode }
XCTAssertFalse(codes.contains(0))
XCTAssertEqual(HelperError(.notAuthorized, "").exitCode, 77)
}
func testSHA256StreamsFiles() throws {
let url = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
defer { try? FileManager.default.removeItem(at: url) }
try Data("abc".utf8).write(to: url)
let digest = try sha256Hex(of: url)
XCTAssertEqual(digest.hex, "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
XCTAssertEqual(digest.sizeBytes, 3)
}
}

View File

@ -0,0 +1,102 @@
import XCTest
@testable import PhotosHelperCore
final class PagingTests: XCTestCase {
func testCursorRoundTripsAndIsURLSafe() throws {
for key in [
SortKey(creationMillis: 1_700_000_000_123, localIdentifier: "9F2A/L0/001"),
SortKey(creationMillis: nil, localIdentifier: "undated+/=?"),
] {
let encoded = ListCursor(key: key).encoded()
XCTAssertFalse(encoded.contains { "+/=".contains($0) }, encoded)
XCTAssertEqual(try ListCursor.decode(encoded).key, key)
}
}
func testCursorRejectsGarbage() {
for value in ["", "!!!", base64URLEncode(Data("{\"v\":2,\"d\":1,\"id\":\"x\"}".utf8)),
base64URLEncode(Data("{\"v\":1,\"d\":1,\"id\":\"\"}".utf8)), base64URLEncode(Data("[]".utf8))]
{
XCTAssertThrowsError(try ListCursor.decode(value), value)
}
}
func testSortKeyPutsUndatedFirstAndBreaksTiesByIdentifier() {
let keys = [
SortKey(creationMillis: 2, localIdentifier: "a"),
SortKey(creationMillis: 1, localIdentifier: "b"),
SortKey(creationMillis: nil, localIdentifier: "z"),
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: nil, localIdentifier: "c"),
]
XCTAssertEqual(
keys.sorted().map { key -> String in
let date = key.creationMillis.map { millis in String(millis) } ?? "-"
return date + key.localIdentifier
},
["-c", "-z", "1a", "1b", "2a"])
}
/// Simulates PhotoKit: sorted by date, ties in arbitrary (here reversed) order.
private func photoKitOrder(_ keys: [SortKey]) -> [SortKey] {
keys.sorted { lhs, rhs in
if lhs.creationMillis == rhs.creationMillis { return lhs.localIdentifier > rhs.localIdentifier }
return lhs < rhs
}
}
private func listAll(_ source: [SortKey], limit: Int) -> [SortKey] {
var cursor: ListCursor?
var listed: [SortKey] = []
var pages = 0
while true {
let page = nextPage(count: source.count, key: { source[$0] }, after: cursor, limit: limit)
XCTAssertLessThanOrEqual(page.indices.count, limit)
listed += page.indices.map { source[$0] }
pages += 1
guard page.hasMore, let last = listed.last else { break }
// The cursor survives a round trip through its encoded form, like the app stores it.
cursor = try? ListCursor.decode(ListCursor(key: last).encoded())
if pages > source.count + 1 { XCTFail("paging did not terminate"); break }
}
return listed
}
func testPagesCoverEverythingOnceInStableOrder() {
var keys: [SortKey] = []
for index in 0..<57 {
// Many equal timestamps (bursts) and a few undated assets.
let millis: Int64? = index % 11 == 0 ? nil : Int64(index / 4)
keys.append(SortKey(creationMillis: millis, localIdentifier: String(format: "id-%03d", index)))
}
let source = photoKitOrder(keys)
for limit in [1, 3, 4, 10, 500] {
XCTAssertEqual(listAll(source, limit: limit), keys.sorted(), "limit \(limit)")
}
}
func testEmptyLibraryHasNoMorePages() {
let page = nextPage(count: 0, key: { _ in fatalError() }, after: nil, limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testCursorAfterLastItemReturnsNothing() {
let source = [SortKey(creationMillis: 1, localIdentifier: "a")]
let page = nextPage(
count: 1, key: { source[$0] }, after: ListCursor(key: source[0]), limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testDeletedCursorItemStillResumesAfterIt() {
let source = [
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: 3, localIdentifier: "c"),
]
let gone = ListCursor(key: SortKey(creationMillis: 2, localIdentifier: "b"))
let page = nextPage(count: 2, key: { source[$0] }, after: gone, limit: 10)
XCTAssertEqual(page.indices, [1])
}
}

View File

@ -0,0 +1,83 @@
import XCTest
@testable import PhotosHelperCore
final class PathGuardTests: XCTestCase {
private var directory: URL!
override func setUpWithError() throws {
directory = FileManager.default.temporaryDirectory.appendingPathComponent(
"photos-helper-\(UUID().uuidString)", isDirectory: true)
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
}
override func tearDownWithError() throws {
try? FileManager.default.removeItem(at: directory)
}
private func resolved() -> URL { directory.resolvingSymlinksInPath().standardizedFileURL }
func testWritesDirectChildrenOfTheResolvedDirectory() throws {
let target = try resolveExportDestination(
directory: directory.path, fileName: "IMG_0001.MOV", temporarySuffix: "t")
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path)
XCTAssertEqual(target.finalURL.lastPathComponent, "IMG_0001.MOV")
XCTAssertEqual(target.temporaryURL.lastPathComponent, ".IMG_0001.MOV.t.partial")
}
func testRejectsRelativeAndTraversingDirectories() {
for path in ["relative/dir", "\(directory.path)/../x", "\(directory.path)/./", "/a/../b", ""] {
XCTAssertThrowsError(
try resolveExportDestination(directory: path, fileName: "a.jpg"), path
) { error in
XCTAssertEqual((error as? HelperError)?.code, .pathRejected)
}
}
}
func testRejectsMissingDirectoryAndFiles() throws {
XCTAssertThrowsError(
try resolveExportDestination(directory: directory.path + "/missing", fileName: "a.jpg"))
let file = directory.appendingPathComponent("file")
try Data().write(to: file)
XCTAssertThrowsError(try resolveExportDestination(directory: file.path, fileName: "a.jpg"))
}
func testHostileFileNamesStayInside() throws {
for name in ["../../etc/passwd", "/etc/passwd", "..", ".hidden", "a/b\\c:d", "\u{0}x", " "] {
let target = try resolveExportDestination(directory: directory.path, fileName: name)
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path, name)
XCTAssertFalse(target.fileName.hasPrefix("."), name)
XCTAssertFalse(target.fileName.contains("/"), name)
}
XCTAssertEqual(sanitizeFileName("../../etc/passwd"), "_.._etc_passwd")
XCTAssertEqual(sanitizeFileName(""), "asset")
}
func testLongNamesKeepTheirExtension() {
let name = sanitizeFileName(String(repeating: "é", count: 300) + ".mov")
XCTAssertLessThanOrEqual(name.utf8.count, 200)
XCTAssertTrue(name.hasSuffix(".mov"))
}
func testNeverOverwritesAnExistingFileOrSymlink() throws {
try Data().write(to: directory.appendingPathComponent("taken.jpg"))
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "taken.jpg")) {
error in XCTAssertEqual((error as? HelperError)?.code, .destinationExists)
}
try FileManager.default.createSymbolicLink(
atPath: directory.appendingPathComponent("link.jpg").path, withDestinationPath: "/nonexistent")
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "link.jpg"))
}
func testSymlinkedDirectoryResolvesToItsTarget() throws {
let link = directory.appendingPathComponent("link", isDirectory: true)
let real = directory.appendingPathComponent("real", isDirectory: true)
try FileManager.default.createDirectory(at: real, withIntermediateDirectories: false)
try FileManager.default.createSymbolicLink(at: link, withDestinationURL: real)
let target = try resolveExportDestination(directory: link.path, fileName: "a.jpg")
XCTAssertEqual(
target.finalURL.deletingLastPathComponent().path,
real.resolvingSymlinksInPath().standardizedFileURL.path)
}
}

View File

@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!-- electron-builder's default hardened-runtime entitlements for Electron. -->
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<!-- Hardened runtime blocks Photos without this, for the app and the bundled photos-helper. -->
<key>com.apple.security.personal-information.photos-library</key>
<true/>
</dict>
</plist>

View File

@ -16,6 +16,8 @@
"release": "pnpm --filter @rakazo/web build && pnpm build && electron-builder --publish never"
},
"dependencies": {
"@orpc/client": "1.15.0",
"@orpc/contract": "1.15.0",
"@rakazo/contracts": "workspace:*",
"electron-updater": "6.8.9"
},
@ -61,7 +63,18 @@
"dmg",
"zip"
],
"notarize": true
"notarize": true,
"entitlements": "native/photos-helper/entitlements.mac.plist",
"entitlementsInherit": "native/photos-helper/entitlements.mac.plist",
"extendInfo": {
"NSPhotoLibraryUsageDescription": "Rakazo adds your photos and videos to the media library on your Rakazo server."
},
"extraResources": [
{
"from": "native/photos-helper/.build/apple/Products/Release/photos-helper",
"to": "bin/photos-helper"
}
]
},
"win": {
"icon": "assets/icon.ico",

View File

@ -0,0 +1,96 @@
import { describe, expect, it } from "vitest";
import {
previewBounds,
previewLoadError,
previewShortcut,
previewUrl,
} from "./browser-preview-policy.js";
describe("previewUrl", () => {
it("accepts only http(s) without credentials", () => {
expect(previewUrl("http://localhost:5173")).toBe("http://localhost:5173/");
expect(previewUrl("https://example.com/a?b=1")).toBe("https://example.com/a?b=1");
for (const value of [
"file:///etc/hosts",
"javascript:alert(1)",
"chrome://settings",
"https://user:fake-password@example.com",
"not a url",
42,
`https://example.com/${"a".repeat(9000)}`,
]) {
expect(previewUrl(value)).toBeNull();
}
});
});
describe("previewBounds", () => {
const content = { width: 1200, height: 800 };
it("rounds and applies the app zoom factor", () => {
expect(previewBounds({ x: 800.4, y: 40.6, width: 399.5, height: 700 }, content)).toEqual({
x: 800,
y: 41,
width: 400,
height: 700,
});
expect(previewBounds({ x: 100, y: 10, width: 200, height: 100 }, content, 1.25)).toEqual({
x: 125,
y: 13,
width: 250,
height: 125,
});
});
it("clamps to the window and rejects empty or malformed rectangles", () => {
expect(previewBounds({ x: -50, y: -10, width: 5000, height: 5000 }, content)).toEqual({
x: 0,
y: 0,
width: 1200,
height: 800,
});
expect(previewBounds({ x: 1300, y: 0, width: 100, height: 100 }, content)).toBeNull();
expect(previewBounds({ x: 0, y: 0, width: 0, height: 100 }, content)).toBeNull();
expect(previewBounds({ x: "1", y: 0, width: 1, height: 1 }, content)).toBeNull();
expect(previewBounds({ x: Number.NaN, y: 0, width: 1, height: 1 }, content)).toBeNull();
expect(previewBounds(null, content)).toBeNull();
});
});
describe("previewShortcut", () => {
const key = (overrides: Partial<Parameters<typeof previewShortcut>[0]>) => ({
type: "keyDown",
key: "r",
control: false,
meta: false,
alt: false,
shift: false,
...overrides,
});
it("maps platform browser shortcuts", () => {
expect(previewShortcut(key({ meta: true }), "darwin")).toBe("reload");
expect(previewShortcut(key({ meta: true, shift: true, key: "R" }), "darwin")).toBe(
"hard-reload",
);
expect(previewShortcut(key({ control: true }), "linux")).toBe("reload");
expect(previewShortcut(key({ meta: true, key: "[" }), "darwin")).toBe("back");
expect(previewShortcut(key({ alt: true, key: "ArrowRight" }), "win32")).toBe("forward");
expect(previewShortcut(key({ key: "F12" }), "linux")).toBe("devtools");
expect(previewShortcut(key({ key: "F5" }), "win32")).toBe("reload");
});
it("leaves typing alone", () => {
expect(previewShortcut(key({}), "darwin")).toBeNull();
expect(previewShortcut(key({ control: true }), "darwin")).toBeNull();
expect(previewShortcut(key({ type: "keyUp", meta: true }), "darwin")).toBeNull();
expect(previewShortcut(key({ key: "ArrowLeft" }), "linux")).toBeNull();
});
});
describe("previewLoadError", () => {
it("ignores superseded navigations", () => {
expect(previewLoadError(-3, "ERR_ABORTED")).toBeNull();
expect(previewLoadError(-102, "ERR_CONNECTION_REFUSED")).toBe("ERR_CONNECTION_REFUSED");
});
});

View File

@ -0,0 +1,76 @@
import type { RakazoDesktopBrowserBounds } from "@rakazo/contracts";
/** Only plain http(s) pages load in the in-app browser; no files, schemes or URL credentials. */
export function previewUrl(value: unknown): string | null {
if (typeof value !== "string" || value.length > 8192) return null;
try {
const url = new URL(value);
if (url.protocol !== "http:" && url.protocol !== "https:") return null;
if (url.username || url.password) return null;
return url.toString();
} catch {
return null;
}
}
/**
* Renderer-reported CSS-pixel bounds, converted to window DIPs and clamped to the content area
* so a compromised renderer cannot place the view outside the panel it reserved.
*/
export function previewBounds(
value: unknown,
content: { width: number; height: number },
zoomFactor = 1,
): RakazoDesktopBrowserBounds | null {
if (!value || typeof value !== "object") return null;
const raw = value as Record<string, unknown>;
const numbers = [raw.x, raw.y, raw.width, raw.height];
if (!numbers.every((n) => typeof n === "number" && Number.isFinite(n))) return null;
const scale = Number.isFinite(zoomFactor) && zoomFactor > 0 ? zoomFactor : 1;
const [x, y, width, height] = (numbers as number[]).map((n) => Math.round(n * scale)) as [
number,
number,
number,
number,
];
const left = Math.min(Math.max(0, x), content.width);
const top = Math.min(Math.max(0, y), content.height);
const right = Math.min(Math.max(left, x + width), content.width);
const bottom = Math.min(Math.max(top, y + height), content.height);
if (right - left < 1 || bottom - top < 1) return null;
return { x: left, y: top, width: right - left, height: bottom - top };
}
export type PreviewShortcut = "reload" | "hard-reload" | "back" | "forward" | "devtools";
/** Browser keyboard shortcuts while focus is inside the page. */
export function previewShortcut(
input: {
type: string;
key: string;
control: boolean;
meta: boolean;
alt: boolean;
shift: boolean;
},
platform: string,
): PreviewShortcut | null {
if (input.type !== "keyDown") return null;
const mod = platform === "darwin" ? input.meta : input.control;
const key = input.key.length === 1 ? input.key.toLowerCase() : input.key;
if (key === "F5") return input.shift ? "hard-reload" : "reload";
if (key === "F12") return "devtools";
if (mod && key === "r") return input.shift ? "hard-reload" : "reload";
if (mod && input.alt && key === "i") return "devtools";
if (platform === "darwin" ? input.meta && !input.alt : input.alt && !input.control) {
if (key === (platform === "darwin" ? "[" : "ArrowLeft")) return "back";
if (key === (platform === "darwin" ? "]" : "ArrowRight")) return "forward";
}
return null;
}
/** Load failures worth showing; an aborted load (-3) is a superseded navigation, not an error. */
export function previewLoadError(errorCode: number, description: string): string | null {
if (errorCode === -3 || errorCode === 0) return null;
return description || `Load failed (${errorCode})`;
}

View File

@ -0,0 +1,189 @@
import type { RakazoDesktopBrowserPreviewState } from "@rakazo/contracts";
import { type BrowserWindow, WebContentsView } from "electron";
import {
previewBounds,
previewLoadError,
previewShortcut,
previewUrl,
} from "./browser-preview-policy.js";
/** Pages share one persistent profile, separate from the app's own session and cookies. */
const PREVIEW_PARTITION = "persist:rakazo-browser";
export const PREVIEW_STATE_CHANNEL = "desktop.browserPreview.state";
/**
* The in-app browser's native page: one WebContentsView laid over the panel the app window
* reserves. It belongs to one app window at a time and is torn down with it.
*/
export class BrowserPreviewController {
private host: BrowserWindow | null = null;
private view: WebContentsView | null = null;
private error: string | null = null;
state(win: BrowserWindow): RakazoDesktopBrowserPreviewState {
const contents = this.host === win ? this.view?.webContents : undefined;
if (!contents || contents.isDestroyed()) {
return {
url: "",
title: "",
loading: false,
canGoBack: false,
canGoForward: false,
error: null,
};
}
const url = contents.getURL();
return {
url,
title: contents.getTitle() === url ? "" : contents.getTitle(),
loading: contents.isLoading(),
canGoBack: contents.navigationHistory.canGoBack(),
canGoForward: contents.navigationHistory.canGoForward(),
error: this.error,
};
}
show(win: BrowserWindow, rawBounds: unknown) {
const view = this.viewFor(win);
const [width, height] = win.getContentSize();
const bounds = previewBounds(
rawBounds,
{ width: width ?? 0, height: height ?? 0 },
win.webContents.getZoomFactor(),
);
if (!bounds) {
view.setVisible(false);
return;
}
view.setBounds(bounds);
view.setVisible(true);
}
hide(win: BrowserWindow) {
if (this.host === win) this.view?.setVisible(false);
}
async navigate(win: BrowserWindow, rawUrl: unknown) {
const url = previewUrl(rawUrl);
if (!url) throw new Error("Only http(s) pages can open here.");
this.error = null;
// Failures surface through did-fail-load as page state, not as a rejected call.
await this.viewFor(win)
.webContents.loadURL(url)
.catch(() => undefined);
}
back(win: BrowserWindow) {
const history = this.contentsFor(win)?.navigationHistory;
if (history?.canGoBack()) history.goBack();
}
forward(win: BrowserWindow) {
const history = this.contentsFor(win)?.navigationHistory;
if (history?.canGoForward()) history.goForward();
}
reload(win: BrowserWindow, ignoreCache = false) {
const contents = this.contentsFor(win);
if (!contents?.getURL()) return;
this.error = null;
if (ignoreCache) contents.reloadIgnoringCache();
else contents.reload();
}
stop(win: BrowserWindow) {
this.contentsFor(win)?.stop();
}
openDevTools(win: BrowserWindow) {
const contents = this.contentsFor(win);
if (contents?.getURL()) contents.openDevTools({ mode: "detach" });
}
private contentsFor(win: BrowserWindow) {
const contents = this.host === win ? this.view?.webContents : undefined;
return contents && !contents.isDestroyed() ? contents : undefined;
}
private viewFor(win: BrowserWindow): WebContentsView {
if (this.host === win && this.view && !this.view.webContents.isDestroyed()) return this.view;
this.destroy();
const view = new WebContentsView({
webPreferences: {
partition: PREVIEW_PARTITION,
sandbox: true,
contextIsolation: true,
nodeIntegration: false,
webSecurity: true,
spellcheck: false,
},
});
view.setVisible(false);
const contents = view.webContents;
const publish = () => {
if (!win.isDestroyed()) win.webContents.send(PREVIEW_STATE_CHANNEL, this.state(win));
};
// One tab: links that want a new window open here instead.
contents.setWindowOpenHandler(({ url }) => {
const target = previewUrl(url);
if (target) void contents.loadURL(target).catch(() => undefined);
return { action: "deny" };
});
contents.on("will-navigate", (event, url) => {
if (!previewUrl(url)) event.preventDefault();
});
contents.on("will-redirect", (event, url) => {
if (!previewUrl(url)) event.preventDefault();
});
contents.on("did-start-loading", publish);
contents.on("did-stop-loading", publish);
contents.on("did-navigate", () => {
this.error = null;
publish();
});
contents.on("did-navigate-in-page", publish);
contents.on("page-title-updated", publish);
contents.on("did-fail-load", (_event, code, description, _url, isMainFrame) => {
if (!isMainFrame) return;
const error = previewLoadError(code, description);
if (error === null) return;
this.error = error;
publish();
});
contents.on("render-process-gone", (_event, details) => {
this.error = details.reason === "clean-exit" ? null : "The page stopped responding.";
publish();
});
contents.on("before-input-event", (event, input) => {
const shortcut = previewShortcut(input, process.platform);
if (!shortcut) return;
event.preventDefault();
if (shortcut === "reload" || shortcut === "hard-reload") {
this.reload(win, shortcut === "hard-reload");
} else if (shortcut === "back") {
this.back(win);
} else if (shortcut === "forward") {
this.forward(win);
} else {
this.openDevTools(win);
}
});
win.contentView.addChildView(view);
win.once("closed", () => {
if (this.host === win) this.destroy();
});
this.host = win;
this.view = view;
return view;
}
private destroy() {
const { host, view } = this;
this.host = null;
this.view = null;
this.error = null;
if (!view) return;
if (host && !host.isDestroyed()) host.contentView.removeChildView(view);
if (!view.webContents.isDestroyed()) view.webContents.close();
}
}

View File

@ -2,7 +2,10 @@ import { randomBytes } from "node:crypto";
import { existsSync } from "node:fs";
import { readFile, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import type { DesktopReachability, DesktopSetup } from "@rakazo/contracts";
import { createORPCClient } from "@orpc/client";
import { RPCLink } from "@orpc/client/fetch";
import type { ContractRouterClient } from "@orpc/contract";
import type { AppContract, DesktopReachability, DesktopSetup } from "@rakazo/contracts";
import { LOCAL_SETTINGS_PAGE } from "@rakazo/contracts/local-settings";
import {
app,
@ -21,6 +24,7 @@ import {
LAUNCH_CHECK_DELAY_MS,
} from "./auto-update.js";
import { openBrowserAuth } from "./browser-auth.js";
import { BrowserPreviewController } from "./browser-preview.js";
import { DOCKER_INSTALL_LINKS, isDesktopSetupLink, runDocker } from "./docker-cli.js";
import { requestLocalSettings } from "./local-settings.js";
import {
@ -31,6 +35,14 @@ import {
stackDir,
stackResourceDir,
} from "./local-stack.js";
import type { MediaIpcEvent } from "./media-ipc.js";
import { registerMediaIpc } from "./media-ipc.js";
import { createPhotosHelper, resolvePhotosHelperBinary } from "./media-photos-helper.js";
import {
createChunkUploader,
createPhotosSyncFileStore,
createPhotosSyncRunner,
} from "./media-photos-sync.js";
import { oauthCallbackFrom } from "./oauth-callback.js";
import {
bundledRendererCandidates,
@ -1129,7 +1141,90 @@ app.whenReady().then(async () => {
);
},
);
const photosBinary = resolvePhotosHelperBinary({
platform: process.platform,
packaged: app.isPackaged,
resourcesPath: process.resourcesPath,
appPath: app.getAppPath(),
exists: existsSync,
});
const photosHelper = photosBinary ? createPhotosHelper(photosBinary) : null;
const mediaIpc = registerMediaIpc({
ipcMain,
platform: process.platform,
helper: photosHelper,
isTrusted: (raw: MediaIpcEvent) => {
const event = raw as Electron.IpcMainInvokeEvent;
return fromMainWindow(event) && event.senderFrame === event.sender.mainFrame;
},
runSync: photosHelper
? createPhotosSyncRunner({
helper: photosHelper,
store: createPhotosSyncFileStore(userDataDir),
tempRoot: app.getPath("temp"),
label: "Apple Photos",
connect: async (spaceId) => {
if (currentTargetUrl === null) throw new Error("No server is open.");
const origin = new URL(currentTargetUrl).origin;
const target = (await resolveSessionForTarget(currentTargetUrl)).value;
const fetchWithSession = (input: string | Request, init?: RequestInit) =>
target.fetch(input, {
...init,
credentials: "include",
bypassCustomProtocolHandlers: true,
});
const api: ContractRouterClient<AppContract> = createORPCClient(
new RPCLink({
url: `${origin}/rpc`,
fetch: (request, init) => {
const headers = new Headers(request.headers);
if (spaceId) headers.set("x-rakazo-space-id", spaceId);
return fetchWithSession(new Request(request, { headers }), init);
},
}),
);
return {
origin,
api,
uploadChunk: createChunkUploader({ origin, spaceId, fetch: fetchWithSession }),
};
},
})
: async () => {
throw new Error("Apple Photos is unavailable.");
},
});
app.on("before-quit", mediaIpc.cancel);
ipcMain.handle("desktop.platform", () => process.platform);
const browserPreview = new BrowserPreviewController();
// The in-app browser answers only the app window's own top frame, never a page inside it.
const previewHost = (event: Electron.IpcMainInvokeEvent) => {
if (!fromMainWindow(event) || event.senderFrame !== event.sender.mainFrame) {
throw new Error("The in-app browser is not available here.");
}
return mainWindow!;
};
ipcMain.handle("desktop.browserPreview.state", (event) =>
browserPreview.state(previewHost(event)),
);
ipcMain.handle("desktop.browserPreview.show", (event, bounds: unknown) =>
browserPreview.show(previewHost(event), bounds),
);
ipcMain.handle("desktop.browserPreview.hide", (event) => browserPreview.hide(previewHost(event)));
ipcMain.handle("desktop.browserPreview.navigate", (event, url: unknown) =>
browserPreview.navigate(previewHost(event), url),
);
ipcMain.handle("desktop.browserPreview.back", (event) => browserPreview.back(previewHost(event)));
ipcMain.handle("desktop.browserPreview.forward", (event) =>
browserPreview.forward(previewHost(event)),
);
ipcMain.handle("desktop.browserPreview.reload", (event) =>
browserPreview.reload(previewHost(event)),
);
ipcMain.handle("desktop.browserPreview.stop", (event) => browserPreview.stop(previewHost(event)));
ipcMain.handle("desktop.browserPreview.openDevTools", (event) =>
browserPreview.openDevTools(previewHost(event)),
);
ipcMain.handle("desktop.files.pickDirectory", async (event) => {
if (!fromMainWindow(event)) return null;
const result = await dialog.showOpenDialog(windowFrom(event)!, {

View File

@ -0,0 +1,161 @@
import { describe, expect, it, vi } from "vitest";
import type { MediaIpcEvent, MediaIpcOptions } from "./media-ipc.js";
import {
MEDIA_PHOTOS_CHANNELS,
outcomeForError,
parseSyncRequest,
registerMediaIpc,
} from "./media-ipc.js";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import type { PhotosSyncResult } from "./media-photos-sync.js";
import { PhotosSyncError } from "./media-photos-sync.js";
type Listener = (event: MediaIpcEvent, ...args: unknown[]) => unknown;
const result: PhotosSyncResult = {
indexed: 2,
created: 2,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: true,
};
function setup(overrides: Partial<MediaIpcOptions> = {}) {
const handlers = new Map<string, Listener>();
const helper = {
authorize: vi.fn(async () => "authorized" as const),
} as unknown as PhotosHelper;
const options: MediaIpcOptions = {
ipcMain: { handle: (channel, listener) => handlers.set(channel, listener) },
platform: "darwin",
helper,
isTrusted: () => true,
runSync: vi.fn(async () => result),
...overrides,
};
const registration = registerMediaIpc(options);
const sent: [string, unknown][] = [];
const event: MediaIpcEvent = {
sender: { send: (channel, payload) => sent.push([channel, payload]), isDestroyed: () => false },
};
const call = (channel: string, ...args: unknown[]) => handlers.get(channel)!(event, ...args);
return { handlers, helper, options, registration, call, sent };
}
describe("media IPC on platforms without Apple Photos", () => {
it.each(["linux", "win32"])(
"reports %s as unavailable and rejects sync and cancel",
async (platform) => {
const { call, options, handlers } = setup({ platform });
expect([...handlers.keys()].sort()).toEqual(
[
MEDIA_PHOTOS_CHANNELS.cancel,
MEDIA_PHOTOS_CHANNELS.status,
MEDIA_PHOTOS_CHANNELS.sync,
].sort(),
);
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: null })).toThrow(/macOS/);
expect(() => call(MEDIA_PHOTOS_CHANNELS.cancel)).toThrow(/macOS/);
expect(options.runSync).not.toHaveBeenCalled();
expect(options.helper!.authorize).not.toHaveBeenCalled();
},
);
it("treats a macOS build without the helper as unavailable", async () => {
const { call } = setup({ helper: null });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow(/macOS/);
});
it("treats a helper that cannot start as unavailable", async () => {
const { call, helper } = setup();
vi.mocked(helper.authorize).mockRejectedValueOnce(new PhotosHelperError("unavailable", "x"));
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
});
});
describe("media IPC on macOS", () => {
it("reads status without prompting", async () => {
const { call, helper } = setup();
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({
available: true,
authorization: "authorized",
syncing: false,
});
expect(helper.authorize).toHaveBeenCalledWith(false);
});
it("runs one sync at a time, forwards progress and resolves with the outcome", async () => {
let finish: (value: PhotosSyncResult) => void = () => {};
const runSync = vi.fn(
({ onProgress }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((resolve) => {
onProgress({ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 });
finish = resolve;
}),
);
const { call, sent } = setup({ runSync });
const first = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
const second = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
expect(second).toBe(first);
expect(runSync).toHaveBeenCalledTimes(1);
expect(runSync.mock.calls[0]?.[0].spaceId).toBe("space_1");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: true });
expect(sent).toEqual([
[
MEDIA_PHOTOS_CHANNELS.progress,
{ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 },
],
]);
finish(result);
await expect(first).resolves.toEqual({ ok: true, result });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: false });
});
it("cancels the running sync", async () => {
const runSync = vi.fn(
({ signal }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((_resolve, reject) => {
signal.addEventListener("abort", () =>
reject(new PhotosSyncError("cancelled", "Sync cancelled.")),
);
}),
);
const { call } = setup({ runSync });
const running = call(MEDIA_PHOTOS_CHANNELS.sync, {});
await call(MEDIA_PHOTOS_CHANNELS.cancel);
await expect(running).resolves.toEqual({
ok: false,
code: "cancelled",
message: "Sync cancelled.",
});
});
it("rejects untrusted senders and malformed requests", async () => {
const { call } = setup({ isTrusted: () => false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow("Not allowed.");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).rejects.toThrow("Not allowed.");
expect(() => parseSyncRequest(null)).toThrow();
expect(() => parseSyncRequest({ spaceId: "../x" })).toThrow();
expect(() => parseSyncRequest({ spaceId: 4 })).toThrow();
expect(parseSyncRequest({})).toEqual({ spaceId: null });
});
it("never forwards unexpected error text to the renderer", () => {
expect(outcomeForError(new Error("/Users/example/secret path"))).toEqual({
ok: false,
code: "failed",
message: "Apple Photos sync failed.",
});
expect(
outcomeForError(
new PhotosHelperError("not_authorized", "Photos access has not been granted."),
),
).toMatchObject({
code: "not_authorized",
});
});
});

View File

@ -0,0 +1,126 @@
import type {
PhotosSourceStatus,
PhotosSyncOutcome,
PhotosSyncProgress,
PhotosSyncResult,
RakazoDesktopMediaPhotos,
} from "@rakazo/contracts";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import { PhotosSyncError } from "./media-photos-sync.js";
/*
* IPC for the Apple Photos source. The renderer only expresses intent (sync, cancel)
* and renders state; the main process owns the helper, the session and the upload.
*/
export const MEDIA_PHOTOS_CHANNELS = {
status: "desktop.media.photos.status",
sync: "desktop.media.photos.sync",
cancel: "desktop.media.photos.cancel",
progress: "desktop.media.photos.progress",
} as const;
export type { PhotosSourceStatus, PhotosSyncOutcome, RakazoDesktopMediaPhotos };
export interface MediaIpcEvent {
sender: { send(channel: string, payload: unknown): void; isDestroyed(): boolean };
}
export interface MediaIpcMain {
handle(channel: string, listener: (event: MediaIpcEvent, ...args: unknown[]) => unknown): void;
}
export interface MediaIpcOptions {
ipcMain: MediaIpcMain;
platform: string;
/** Null when the helper binary is missing (unbuilt dev checkout). */
helper: PhotosHelper | null;
/** Same gate as the other app-window handlers: main window, main frame. */
isTrusted: (event: MediaIpcEvent) => boolean;
runSync: (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}) => Promise<PhotosSyncResult>;
}
const UNAVAILABLE = "Apple Photos is only available in the macOS app.";
const SPACE_ID = /^[A-Za-z0-9_-]{1,200}$/;
export function parseSyncRequest(value: unknown): { spaceId: string | null } {
if (typeof value !== "object" || value === null) throw new Error("Invalid sync request.");
const spaceId = (value as { spaceId?: unknown }).spaceId ?? null;
if (spaceId !== null && (typeof spaceId !== "string" || !SPACE_ID.test(spaceId))) {
throw new Error("Invalid sync request.");
}
return { spaceId };
}
/** Registers the handlers; `cancel()` stops a running sync, e.g. before quit. */
export function registerMediaIpc(options: MediaIpcOptions): { cancel: () => void } {
const available = options.platform === "darwin" && options.helper !== null;
let running: { controller: AbortController; done: Promise<PhotosSyncOutcome> } | null = null;
const requireTrusted = (event: MediaIpcEvent) => {
if (!options.isTrusted(event)) throw new Error("Not allowed.");
};
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.status, async (event) => {
requireTrusted(event);
if (!available) return { available: false } satisfies PhotosSourceStatus;
try {
const authorization = await options.helper!.authorize(false);
return { available: true, authorization, syncing: running !== null };
} catch (error) {
if (error instanceof PhotosHelperError && error.code === "unavailable") {
return { available: false } satisfies PhotosSourceStatus;
}
throw error;
}
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.sync, (event, request: unknown) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
const { spaceId } = parseSyncRequest(request);
// A second click joins the running sync instead of starting a parallel one.
if (running) return running.done;
const controller = new AbortController();
const done = options
.runSync({
spaceId,
signal: controller.signal,
onProgress: (progress) => {
if (!event.sender.isDestroyed()) {
event.sender.send(MEDIA_PHOTOS_CHANNELS.progress, progress);
}
},
})
.then(
(result): PhotosSyncOutcome => ({ ok: true, result }),
(error: unknown): PhotosSyncOutcome => outcomeForError(error),
)
.finally(() => {
if (running?.controller === controller) running = null;
});
running = { controller, done };
return done;
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.cancel, (event) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
running?.controller.abort();
});
return { cancel: () => running?.controller.abort() };
}
/** Only our own fixed messages reach the renderer; anything else is generic. */
export function outcomeForError(error: unknown): PhotosSyncOutcome {
if (error instanceof PhotosSyncError || error instanceof PhotosHelperError) {
return { ok: false, code: error.code, message: error.message };
}
return { ok: false, code: "failed", message: "Apple Photos sync failed." };
}

View File

@ -0,0 +1,437 @@
import { EventEmitter } from "node:events";
import { mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { PassThrough } from "node:stream";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { HelperProcess, SpawnHelper } from "./media-photos-helper.js";
import {
changesArgs,
createPhotosHelper,
exportArgs,
helperEnv,
listArgs,
PhotosHelperError,
parseAssetOutput,
parseAuthorizeOutput,
parseChangesOutput,
parseExportOutput,
parseHelperError,
parseListOutput,
parseProgressLine,
photosHelperCandidates,
resolvePhotosHelperBinary,
runPhotosHelper,
} from "./media-photos-helper.js";
const item = (overrides: Record<string, unknown> = {}) => ({
externalId: "9F2A61B0-0000-4000-8000-000000000001/L0/001",
kind: "video",
capturedAt: "2024-01-02T03:04:05.500Z",
durationMs: 12_346,
width: 1920,
height: 1080,
fps: null,
gpsLat: 52.52,
gpsLon: 13.4,
deviceModel: null,
albumNames: ["Trip"],
isFavorite: true,
fileName: "IMG_0001.MOV",
...overrides,
});
const listJson = (items: unknown[], nextCursor: unknown = null, changeToken: unknown = "tok") =>
JSON.stringify({ items, nextCursor, changeToken });
function expectCode(run: () => unknown, code: string) {
try {
run();
} catch (error) {
expect(error).toBeInstanceOf(PhotosHelperError);
expect((error as PhotosHelperError).code).toBe(code);
return;
}
throw new Error("expected a PhotosHelperError");
}
describe("photos helper output parsing", () => {
it("accepts a list page that matches the shared item contract", () => {
const page = parseListOutput(
listJson([item(), item({ kind: "photo", durationMs: null })], "c1"),
2,
);
expect(page.nextCursor).toBe("c1");
expect(page.changeToken).toBe("tok");
expect(page.items).toHaveLength(2);
expect(page.items[0]).toEqual(item());
});
it("accepts the last page of an empty library", () => {
expect(parseListOutput(listJson([], null, null), 500)).toEqual({
items: [],
nextCursor: null,
changeToken: null,
});
});
it.each([
["not JSON", "items: []"],
["an array document", "[]"],
["truncated JSON", '{"items":[{"externalId":'],
["missing items", JSON.stringify({ nextCursor: null, changeToken: null })],
["an item missing a key", listJson([{ ...item(), width: undefined }])],
["an unknown kind", listJson([item({ kind: "audio" })])],
["out-of-range GPS", listJson([item({ gpsLat: 91 })])],
["a non-UTC-parsable date", listJson([item({ capturedAt: "yesterday" })])],
["a negative duration", listJson([item({ durationMs: -1 })])],
["an empty external id", listJson([item({ externalId: "" })])],
["an empty cursor", listJson([item()], "")],
["a numeric cursor", listJson([item()], 3)],
["too many items", listJson([item(), item(), item()])],
])("rejects %s", (_name, text) => {
expectCode(() => parseListOutput(text, 2), "invalid_output");
});
it("parses authorization states and rejects unknown ones", () => {
expect(parseAuthorizeOutput('{"status":"limited"}')).toBe("limited");
expect(parseAuthorizeOutput('{"status":"notDetermined"}')).toBe("notDetermined");
expectCode(() => parseAuthorizeOutput('{"status":"maybe"}'), "invalid_output");
expectCode(() => parseAuthorizeOutput(""), "invalid_output");
});
it("parses asset details", () => {
const detail = parseAssetOutput(
JSON.stringify({
item: item(),
modifiedAt: "2024-02-01T00:00:00.000Z",
hidden: false,
resources: [{ type: "video", fileName: "IMG_0001.MOV", uti: "com.apple.quicktime-movie" }],
}),
);
expect(detail.resources[0]?.type).toBe("video");
expectCode(
() => parseAssetOutput(JSON.stringify({ item: item(), modifiedAt: null, resources: [] })),
"invalid_output",
);
});
it("parses changes and requires a token unless a full relisting is needed", () => {
expect(
parseChangesOutput(
JSON.stringify({
token: "t2",
changed: [item()],
deleted: ["x/L0/001"],
fullResyncRequired: false,
}),
),
).toMatchObject({ token: "t2", deleted: ["x/L0/001"], fullResyncRequired: false });
expect(
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: true }),
).fullResyncRequired,
).toBe(true);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: false }),
),
"invalid_output",
);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: "t", changed: [], deleted: [7], fullResyncRequired: false }),
),
"invalid_output",
);
});
it("maps helper error lines to codes", () => {
const stderr =
'{"progress":0.5}\n{"error":{"code":"not_authorized","message":"Photos access has not been granted."}}\n';
const error = parseHelperError(stderr, 77);
expect(error.code).toBe("not_authorized");
expect(error.message).toBe("Photos access has not been granted.");
expect(parseHelperError('{"error":{"code":"internal","message":"x"}}', 70).code).toBe("failed");
expect(parseHelperError("Segmentation fault", 139).message).toContain("139");
expect(parseHelperError('{"error":"oops"}', 1).code).toBe("failed");
});
it("reads progress lines only", () => {
expect(parseProgressLine('{"progress":0.42}')).toBe(0.42);
expect(parseProgressLine('{"progress":2}')).toBeNull();
expect(parseProgressLine('{"error":{"code":"usage"}}')).toBeNull();
expect(parseProgressLine('{"progress":')).toBeNull();
});
});
describe("photos helper export result", () => {
let dir: string;
let other: string;
beforeEach(async () => {
dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-export-"));
other = await mkdtemp(path.join(tmpdir(), "rakazo-photos-other-"));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
await rm(other, { recursive: true, force: true });
});
const result = (file: string, fileName = path.basename(file)) =>
JSON.stringify({ path: file, fileName, sizeBytes: 3, sha256: "a".repeat(64) });
it("accepts a file directly inside the export folder", async () => {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
await expect(parseExportOutput(result(file), dir)).resolves.toMatchObject({
fileName: "IMG_0001.MOV",
sizeBytes: 3,
});
});
it("rejects files outside the export folder, including through symlinks", async () => {
const outside = path.join(other, "secret.txt");
await writeFile(outside, "abc");
await expect(parseExportOutput(result(outside), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const link = path.join(dir, "link.MOV");
await symlink(outside, link);
await expect(parseExportOutput(result(link), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const nested = path.join(dir, "..", path.basename(other), "secret.txt");
await expect(parseExportOutput(result(nested), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
it("rejects malformed results", async () => {
const file = path.join(dir, "a.jpg");
await writeFile(file, "abc");
for (const text of [
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: 3, sha256: "XYZ" }),
JSON.stringify({ path: "a.jpg", fileName: "a.jpg", sizeBytes: 3, sha256: "a".repeat(64) }),
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: -1, sha256: "a".repeat(64) }),
JSON.stringify({
path: path.join(dir, "missing.jpg"),
fileName: "missing.jpg",
sizeBytes: 3,
sha256: "a".repeat(64),
}),
]) {
await expect(parseExportOutput(text, dir)).rejects.toMatchObject({ code: "invalid_output" });
}
await expect(parseExportOutput(result(file, "b.jpg"), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
});
describe("photos helper argv", () => {
it("builds fixed argv arrays", () => {
expect(listArgs({ cursor: null, limit: 500 })).toEqual(["list", "--limit", "500"]);
expect(listArgs({ cursor: "abc_-", limit: 10, since: "2024-01-01T00:00:00Z" })).toEqual([
"list",
"--limit",
"10",
"--cursor",
"abc_-",
"--since",
"2024-01-01T00:00:00Z",
]);
expect(exportArgs("ID/L0/001", "/tmp/rakazo-photos-1")).toEqual([
"export",
"ID/L0/001",
"--dest",
"/tmp/rakazo-photos-1",
"--original",
]);
expect(changesArgs("tok")).toEqual(["changes", "--token", "tok"]);
});
it("keeps shell syntax as inert data and rejects option-like values", () => {
// No shell runs, so metacharacters stay one literal argument.
expect(exportArgs("$(rm -rf ~);`x`", "/tmp/a b")[1]).toBe("$(rm -rf ~);`x`");
expectCode(() => exportArgs("--dest", "/tmp"), "usage");
expectCode(() => listArgs({ cursor: "--limit", limit: 1 }), "usage");
expectCode(() => changesArgs("a\nb"), "usage");
expectCode(() => listArgs({ cursor: null, limit: 501 }), "usage");
expectCode(() => exportArgs("ID", "relative/dir"), "path_rejected");
expectCode(() => exportArgs("ID", "/tmp/../etc"), "path_rejected");
});
it("passes only the environment PhotoKit needs", () => {
expect(
helperEnv({ HOME: "/Users/example", TMPDIR: "/tmp", OPENAI_API_KEY: "secret", PATH: "/bin" }),
).toEqual({ HOME: "/Users/example", TMPDIR: "/tmp" });
});
});
describe("photos helper binary", () => {
it("is only available on macOS", () => {
const options = {
packaged: true,
resourcesPath: "/App/Resources",
appPath: "/app",
exists: () => true,
};
expect(resolvePhotosHelperBinary({ ...options, platform: "linux" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "win32" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "darwin" })).toBe(
"/App/Resources/bin/photos-helper",
);
});
it("falls back to the development build and reports a missing build", () => {
const options = {
platform: "darwin",
packaged: false,
resourcesPath: "/r",
appPath: "/repo/apps/desktop",
};
expect(photosHelperCandidates(options)).toEqual([
"/repo/apps/desktop/native/photos-helper/.build/apple/Products/Release/photos-helper",
"/repo/apps/desktop/native/photos-helper/.build/release/photos-helper",
]);
expect(
resolvePhotosHelperBinary({
...options,
exists: (file) => file.endsWith(".build/release/photos-helper"),
}),
).toBe("/repo/apps/desktop/native/photos-helper/.build/release/photos-helper");
expect(resolvePhotosHelperBinary({ ...options, exists: () => false })).toBeNull();
});
});
class FakeProcess extends EventEmitter implements HelperProcess {
stdout = new PassThrough();
stderr = new PassThrough();
kill = vi.fn(() => {
queueMicrotask(() => this.emit("close", null));
return true;
});
exit(code: number, stdout = "", stderr = "") {
if (stdout) this.stdout.write(stdout);
if (stderr) this.stderr.write(stderr);
setImmediate(() => this.emit("close", code));
}
}
function fakeSpawn() {
const processes: FakeProcess[] = [];
const spawn = vi.fn<SpawnHelper>(() => {
const child = new FakeProcess();
processes.push(child);
return child;
});
return { spawn, processes };
}
describe("photos helper process", () => {
it("spawns the binary with argv and no shell", async () => {
const { spawn, processes } = fakeSpawn();
const helper = createPhotosHelper("/App/Resources/bin/photos-helper", spawn);
const pending = helper.list({ cursor: null, limit: 2 });
processes[0]!.exit(0, listJson([item()]));
await expect(pending).resolves.toMatchObject({ items: [item()] });
expect(spawn).toHaveBeenCalledWith(
"/App/Resources/bin/photos-helper",
["list", "--limit", "2"],
expect.objectContaining({ shell: false, stdio: ["ignore", "pipe", "pipe"] }),
);
});
it("maps a failing exit to the helper's error code", async () => {
const { spawn, processes } = fakeSpawn();
const pending = createPhotosHelper("/bin/photos-helper", spawn).asset("ID/L0/001");
processes[0]!.exit(
66,
"",
'{"error":{"code":"not_found","message":"No photo or video with that identifier."}}\n',
);
await expect(pending).rejects.toMatchObject({ code: "not_found" });
});
it("reports export progress and validates the result", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-run-"));
try {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
const { spawn, processes } = fakeSpawn();
const progress: number[] = [];
const pending = createPhotosHelper("/bin/photos-helper", spawn).exportOriginal("ID", dir, {
onProgress: (value) => progress.push(value),
});
const child = processes[0]!;
child.stderr.write('{"progress":0}\n{"progress":0.5');
child.stderr.write('}\nnoise\n{"progress":1}\n');
child.exit(
0,
JSON.stringify({
path: file,
fileName: "IMG_0001.MOV",
sizeBytes: 3,
sha256: "b".repeat(64),
}),
);
await expect(pending).resolves.toMatchObject({ sizeBytes: 3 });
expect(progress).toEqual([0, 0.5, 1]);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("kills the helper on abort and on timeout", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
const aborted = runPhotosHelper("/bin/h", ["list"], {
spawn,
timeoutMs: 60_000,
signal: controller.signal,
});
controller.abort();
await expect(aborted).rejects.toMatchObject({ code: "cancelled" });
expect(processes[0]!.kill).toHaveBeenCalledWith("SIGTERM");
const timedOut = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 5 });
await expect(timedOut).rejects.toMatchObject({ code: "timeout" });
expect(processes[1]!.kill).toHaveBeenCalled();
});
it("fails a silent export through the idle limit", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["export"], {
spawn,
timeoutMs: 60_000,
idleTimeoutMs: 5,
});
await expect(pending).rejects.toMatchObject({ code: "timeout" });
expect(processes[0]!.kill).toHaveBeenCalled();
});
it("does not start when already aborted and reports spawn failures", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
controller.abort();
await expect(
runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000, signal: controller.signal }),
).rejects.toMatchObject({ code: "cancelled" });
expect(spawn).not.toHaveBeenCalled();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.emit("error", new Error("ENOENT"));
await expect(pending).rejects.toMatchObject({ code: "unavailable" });
});
it("rejects oversized output", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.stdout.write(Buffer.alloc(33 * 1024 * 1024));
await expect(pending).rejects.toMatchObject({ code: "invalid_output" });
});
});

View File

@ -0,0 +1,512 @@
import { spawn as nodeSpawn } from "node:child_process";
import { realpath } from "node:fs/promises";
import path from "node:path";
import type { Readable } from "node:stream";
import type { ClientMediaItem, PhotosAuthorization } from "@rakazo/contracts";
import { ClientMediaItemSchema } from "@rakazo/contracts";
/*
* Client for the macOS `photos-helper` CLI (apps/desktop/native/photos-helper). The
* helper runs with a fixed argv and no shell; every byte it prints is validated before
* the app uses it, so a malformed or unexpected answer fails instead of being guessed at.
*/
export type { PhotosAuthorization };
export interface PhotosListPage {
items: ClientMediaItem[];
nextCursor: string | null;
/** Library change token from before this page was read. */
changeToken: string | null;
}
export interface PhotosAssetDetail {
item: ClientMediaItem;
modifiedAt: string | null;
hidden: boolean;
resources: { type: string; fileName: string; uti: string }[];
}
export interface PhotosChanges {
token: string | null;
changed: ClientMediaItem[];
deleted: string[];
fullResyncRequired: boolean;
}
export interface PhotosExport {
path: string;
fileName: string;
sizeBytes: number;
sha256: string;
}
export type PhotosHelperErrorCode =
| "unavailable"
| "usage"
| "not_authorized"
| "not_found"
| "path_rejected"
| "destination_exists"
| "network"
| "export_failed"
| "invalid_output"
| "timeout"
| "cancelled"
| "failed";
const HELPER_CODES = new Set<PhotosHelperErrorCode>([
"usage",
"not_authorized",
"not_found",
"path_rejected",
"destination_exists",
"network",
"export_failed",
]);
export class PhotosHelperError extends Error {
constructor(
readonly code: PhotosHelperErrorCode,
message: string,
) {
super(message);
this.name = "PhotosHelperError";
}
}
export const PHOTOS_HELPER_NAME = "photos-helper";
/** Matches the contract's batch limit for `media.sources.pushItems`. */
export const PHOTOS_MAX_PAGE = 500;
/**
* Packaged apps ship the universal helper in Resources/bin; development uses the
* SwiftPM output (universal `--arch` build first, then the host-arch build).
*/
export function photosHelperCandidates(options: {
packaged: boolean;
resourcesPath: string;
appPath: string;
}): string[] {
if (options.packaged) return [path.join(options.resourcesPath, "bin", PHOTOS_HELPER_NAME)];
const build = path.join(options.appPath, "native", "photos-helper", ".build");
return [
path.join(build, "apple", "Products", "Release", PHOTOS_HELPER_NAME),
path.join(build, "release", PHOTOS_HELPER_NAME),
];
}
/** Null off macOS or when the helper was not built; the Apple Photos source is then hidden. */
export function resolvePhotosHelperBinary(options: {
platform: string;
packaged: boolean;
resourcesPath: string;
appPath: string;
exists: (file: string) => boolean;
}): string | null {
if (options.platform !== "darwin") return null;
return photosHelperCandidates(options).find((candidate) => options.exists(candidate)) ?? null;
}
// ---------------------------------------------------------------------------
// argv
// ---------------------------------------------------------------------------
export function authorizeArgs(prompt: boolean): string[] {
return prompt ? ["authorize"] : ["authorize", "--no-prompt"];
}
export function listArgs(request: {
cursor: string | null;
limit: number;
since?: string | null;
}): string[] {
if (!Number.isInteger(request.limit) || request.limit < 1 || request.limit > PHOTOS_MAX_PAGE) {
throw new PhotosHelperError("usage", "List limit is out of range.");
}
const args = ["list", "--limit", String(request.limit)];
if (request.cursor !== null) args.push("--cursor", opaqueArgument(request.cursor));
if (request.since) args.push("--since", opaqueArgument(request.since));
return args;
}
export function assetArgs(localIdentifier: string): string[] {
return ["asset", identifierArgument(localIdentifier)];
}
export function exportArgs(localIdentifier: string, destination: string): string[] {
if (!path.isAbsolute(destination) || destination.split(/[\\/]/).includes("..")) {
throw new PhotosHelperError("path_rejected", "Export directory must be absolute.");
}
return ["export", identifierArgument(localIdentifier), "--dest", destination, "--original"];
}
export function changesArgs(token: string): string[] {
return ["changes", "--token", opaqueArgument(token)];
}
/** Identifiers come from the helper itself; a leading dash would read as an option. */
function identifierArgument(value: string): string {
if (value === "" || value.length > 200 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid Photos identifier.");
}
return value;
}
function opaqueArgument(value: string): string {
if (value === "" || value.length > 4096 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid helper argument.");
}
return value;
}
function hasControl(value: string): boolean {
for (const char of value) {
const code = char.codePointAt(0) ?? 0;
if (code < 0x20 || code === 0x7f) return true;
}
return false;
}
// ---------------------------------------------------------------------------
// Output parsing
// ---------------------------------------------------------------------------
function invalid(what: string): never {
throw new PhotosHelperError("invalid_output", `Photos helper returned invalid ${what}.`);
}
function parseDocument(text: string): Record<string, unknown> {
let value: unknown;
try {
value = JSON.parse(text);
} catch {
invalid("JSON");
}
if (typeof value !== "object" || value === null || Array.isArray(value)) invalid("JSON");
return value as Record<string, unknown>;
}
function nullableString(value: unknown, what: string, max = 4096): string | null {
if (value === null) return null;
if (typeof value !== "string" || value === "" || value.length > max) invalid(what);
return value;
}
/** Each item must satisfy the shared contract schema exactly as the server will parse it. */
function parseItems(value: unknown, what: string): ClientMediaItem[] {
if (!Array.isArray(value) || value.length > 10_000) invalid(what);
return value.map((entry) => {
if (typeof entry !== "object" || entry === null) invalid(what);
const parsed = ClientMediaItemSchema.safeParse(entry);
if (!parsed.success) invalid(what);
return parsed.data;
});
}
const AUTHORIZATION = new Set<PhotosAuthorization>([
"authorized",
"limited",
"denied",
"restricted",
"notDetermined",
]);
export function parseAuthorizeOutput(text: string): PhotosAuthorization {
const status = parseDocument(text).status;
if (typeof status !== "string" || !AUTHORIZATION.has(status as PhotosAuthorization)) {
invalid("authorization status");
}
return status as PhotosAuthorization;
}
export function parseListOutput(text: string, limit: number): PhotosListPage {
const document = parseDocument(text);
const items = parseItems(document.items, "list items");
if (items.length > limit) invalid("list page size");
const nextCursor = nullableString(document.nextCursor, "list cursor");
const changeToken = nullableString(document.changeToken, "change token");
return { items, nextCursor, changeToken };
}
export function parseAssetOutput(text: string): PhotosAssetDetail {
const document = parseDocument(text);
const [item] = parseItems([document.item], "asset");
const modifiedAt = nullableString(document.modifiedAt, "asset date", 64);
if (modifiedAt !== null && Number.isNaN(Date.parse(modifiedAt))) invalid("asset date");
if (typeof document.hidden !== "boolean") invalid("asset");
if (!Array.isArray(document.resources) || document.resources.length > 50) invalid("resources");
const resources = document.resources.map((resource) => {
if (typeof resource !== "object" || resource === null) invalid("resources");
const { type, fileName, uti } = resource as Record<string, unknown>;
if (typeof type !== "string" || typeof fileName !== "string" || typeof uti !== "string") {
invalid("resources");
}
return { type, fileName, uti };
});
return { item: item!, modifiedAt, hidden: document.hidden, resources };
}
export function parseChangesOutput(text: string): PhotosChanges {
const document = parseDocument(text);
const token = nullableString(document.token, "change token");
if (typeof document.fullResyncRequired !== "boolean") invalid("changes");
const changed = parseItems(document.changed, "changed items");
if (!Array.isArray(document.deleted) || document.deleted.length > 10_000) invalid("changes");
const deleted = document.deleted.map((id) => {
if (typeof id !== "string" || id === "" || id.length > 200) invalid("deleted ids");
return id;
});
if (!document.fullResyncRequired && token === null) invalid("change token");
return { token, changed, deleted, fullResyncRequired: document.fullResyncRequired };
}
/**
* Besides the shape, the file must be a direct child of the directory the app chose,
* after resolving symlinks, so a confused helper cannot point the upload elsewhere.
*/
export async function parseExportOutput(text: string, destination: string): Promise<PhotosExport> {
const document = parseDocument(text);
const { path: file, fileName, sizeBytes, sha256 } = document;
if (
typeof file !== "string" ||
!path.isAbsolute(file) ||
typeof fileName !== "string" ||
fileName === "" ||
typeof sizeBytes !== "number" ||
!Number.isSafeInteger(sizeBytes) ||
sizeBytes < 0 ||
typeof sha256 !== "string" ||
!/^[0-9a-f]{64}$/.test(sha256)
) {
invalid("export result");
}
let real: string;
let realDestination: string;
try {
[real, realDestination] = await Promise.all([realpath(file), realpath(destination)]);
} catch {
invalid("export path");
}
if (path.dirname(real) !== realDestination || path.basename(real) !== fileName) {
throw new PhotosHelperError("path_rejected", "Photos helper wrote outside the export folder.");
}
return { path: real, fileName, sizeBytes, sha256 };
}
/** The last `{"error":{code,message}}` line on stderr; unknown codes become `failed`. */
export function parseHelperError(stderr: string, exitCode: number | null): PhotosHelperError {
const lines = stderr.split("\n").reverse();
for (const line of lines) {
if (!line.startsWith("{")) continue;
try {
const error = (JSON.parse(line) as { error?: unknown }).error as
| { code?: unknown; message?: unknown }
| undefined;
if (typeof error?.code !== "string") continue;
const code = HELPER_CODES.has(error.code as PhotosHelperErrorCode)
? (error.code as PhotosHelperErrorCode)
: "failed";
const message =
typeof error.message === "string" && error.message.length <= 500
? error.message
: "Photos helper failed.";
return new PhotosHelperError(code, message);
} catch {}
}
return new PhotosHelperError("failed", `Photos helper exited with status ${exitCode ?? "?"}.`);
}
/** `{"progress":0.42}` lines on stderr; anything else is ignored. */
export function parseProgressLine(line: string): number | null {
if (!line.startsWith('{"progress"')) return null;
try {
const value = (JSON.parse(line) as { progress?: unknown }).progress;
return typeof value === "number" && value >= 0 && value <= 1 ? value : null;
} catch {
return null;
}
}
// ---------------------------------------------------------------------------
// Process
// ---------------------------------------------------------------------------
/** The subset of ChildProcess the runner needs, so tests can supply a fake. */
export interface HelperProcess {
stdout: Readable | null;
stderr: Readable | null;
kill(signal?: NodeJS.Signals): boolean;
on(event: "error", listener: (error: Error) => void): unknown;
on(event: "close", listener: (code: number | null) => void): unknown;
}
export type SpawnHelper = (
binary: string,
args: string[],
options: { shell: false; env: Record<string, string>; stdio: ["ignore", "pipe", "pipe"] },
) => HelperProcess;
export interface RunOptions {
signal?: AbortSignal;
/** Hard limit for the whole run. */
timeoutMs: number;
/** Fails when stderr progress stops for this long (exports only). */
idleTimeoutMs?: number;
onProgress?: (fraction: number) => void;
}
const MAX_STDOUT_BYTES = 32 * 1024 * 1024;
const MAX_STDERR_BYTES = 1024 * 1024;
const ENV_ALLOWLIST = ["HOME", "USER", "LOGNAME", "TMPDIR", "LANG", "LC_ALL"];
/** PhotoKit needs the user's home; nothing else from the app environment is passed on. */
export function helperEnv(env: NodeJS.ProcessEnv): Record<string, string> {
const result: Record<string, string> = {};
for (const name of ENV_ALLOWLIST) {
const value = env[name];
if (value) result[name] = value;
}
return result;
}
export const defaultSpawnHelper: SpawnHelper = (binary, args, options) =>
nodeSpawn(binary, args, options);
export function runPhotosHelper(
binary: string,
args: string[],
options: RunOptions & { spawn?: SpawnHelper; env?: NodeJS.ProcessEnv },
): Promise<string> {
return new Promise((resolve, reject) => {
if (options.signal?.aborted) {
reject(new PhotosHelperError("cancelled", "Cancelled."));
return;
}
const child = (options.spawn ?? defaultSpawnHelper)(binary, args, {
shell: false,
env: helperEnv(options.env ?? process.env),
stdio: ["ignore", "pipe", "pipe"],
});
const stdout: Buffer[] = [];
let stdoutBytes = 0;
let stderr = "";
let pendingLine = "";
let settled = false;
let idleTimer: NodeJS.Timeout | undefined;
const finish = (error: PhotosHelperError | null, output = "") => {
if (settled) return;
settled = true;
clearTimeout(timer);
clearTimeout(idleTimer);
options.signal?.removeEventListener("abort", abort);
if (error) reject(error);
else resolve(output);
};
const stop = (error: PhotosHelperError) => {
child.kill("SIGTERM");
finish(error);
};
const abort = () => stop(new PhotosHelperError("cancelled", "Cancelled."));
const timer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos helper timed out.")),
options.timeoutMs,
);
timer.unref?.();
const armIdle = () => {
if (options.idleTimeoutMs === undefined) return;
clearTimeout(idleTimer);
idleTimer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos export stalled.")),
options.idleTimeoutMs,
);
idleTimer.unref?.();
};
armIdle();
options.signal?.addEventListener("abort", abort, { once: true });
child.stdout?.on("data", (chunk: Buffer) => {
stdoutBytes += chunk.byteLength;
if (stdoutBytes > MAX_STDOUT_BYTES) {
stop(new PhotosHelperError("invalid_output", "Photos helper output is too large."));
return;
}
stdout.push(chunk);
});
child.stderr?.on("data", (chunk: Buffer) => {
const text = chunk.toString("utf8");
if (stderr.length < MAX_STDERR_BYTES) stderr += text;
const lines = (pendingLine + text).split("\n");
pendingLine = lines.pop() ?? "";
for (const line of lines) {
const progress = parseProgressLine(line.trim());
if (progress !== null) {
armIdle();
options.onProgress?.(progress);
}
}
});
child.on("error", () => {
finish(new PhotosHelperError("unavailable", "Photos helper could not be started."));
});
child.on("close", (code) => {
if (code === 0) finish(null, Buffer.concat(stdout).toString("utf8"));
else finish(parseHelperError(stderr + pendingLine, code));
});
});
}
const QUERY_TIMEOUT_MS = 2 * 60_000;
/** The system permission dialog waits on the person. */
const AUTHORIZE_TIMEOUT_MS = 10 * 60_000;
/** Large iCloud videos take long; a stalled download fails sooner through the idle limit. */
const EXPORT_TIMEOUT_MS = 2 * 60 * 60_000;
const EXPORT_IDLE_TIMEOUT_MS = 5 * 60_000;
export interface PhotosHelper {
authorize(prompt: boolean, signal?: AbortSignal): Promise<PhotosAuthorization>;
list(
request: { cursor: string | null; limit: number; since?: string | null },
signal?: AbortSignal,
): Promise<PhotosListPage>;
asset(localIdentifier: string, signal?: AbortSignal): Promise<PhotosAssetDetail>;
exportOriginal(
localIdentifier: string,
destination: string,
options?: { signal?: AbortSignal; onProgress?: (fraction: number) => void },
): Promise<PhotosExport>;
changes(token: string, signal?: AbortSignal): Promise<PhotosChanges>;
}
export function createPhotosHelper(binary: string, spawn?: SpawnHelper): PhotosHelper {
const run = (args: string[], options: RunOptions) =>
runPhotosHelper(binary, args, { ...options, spawn });
return {
async authorize(prompt, signal) {
const timeoutMs = prompt ? AUTHORIZE_TIMEOUT_MS : QUERY_TIMEOUT_MS;
return parseAuthorizeOutput(await run(authorizeArgs(prompt), { signal, timeoutMs }));
},
async list(request, signal) {
const output = await run(listArgs(request), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseListOutput(output, request.limit);
},
async asset(localIdentifier, signal) {
const output = await run(assetArgs(localIdentifier), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseAssetOutput(output);
},
async exportOriginal(localIdentifier, destination, options = {}) {
const output = await run(exportArgs(localIdentifier, destination), {
signal: options.signal,
onProgress: options.onProgress,
timeoutMs: EXPORT_TIMEOUT_MS,
idleTimeoutMs: EXPORT_IDLE_TIMEOUT_MS,
});
return parseExportOutput(output, destination);
},
async changes(token, signal) {
const output = await run(changesArgs(token), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseChangesOutput(output);
},
};
}

View File

@ -0,0 +1,655 @@
import { createHash } from "node:crypto";
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { ClientMediaItem, MediaAsset, MediaUpload } from "@rakazo/contracts";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { PhotosHelper } from "./media-photos-helper.js";
import type { MediaSyncApi, PhotosSyncState, PhotosSyncStore } from "./media-photos-sync.js";
import {
createChunkUploader,
createPhotosSyncFileStore,
parsePhotosSyncState,
runPool,
syncApplePhotos,
} from "./media-photos-sync.js";
const sha = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
function libraryItem(index: number): ClientMediaItem {
return {
externalId: `ASSET-${String(index).padStart(5, "0")}/L0/001`,
kind: index % 3 === 0 ? "video" : "photo",
capturedAt: new Date(Date.UTC(2024, 0, 1) + index * 60_000).toISOString(),
durationMs: index % 3 === 0 ? 5000 : null,
width: 1920,
height: 1080,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
fileName: `IMG_${index}.JPG`,
};
}
/** Pages a synthetic library the way the Swift helper does: opaque cursor = next index. */
function fakeHelper(options: { size: number; contents?: (id: string) => string }) {
const library = Array.from({ length: options.size }, (_, index) => libraryItem(index));
const exports: string[] = [];
let active = 0;
const stats = { maxActiveExports: 0 };
const helper: PhotosHelper & { library: ClientMediaItem[] } = {
library,
authorize: vi.fn(async () => "authorized" as const),
list: vi.fn(async ({ cursor, limit }) => {
const start = cursor === null ? 0 : Number(cursor);
const items = library.slice(start, start + limit);
const next = start + limit < library.length ? String(start + limit) : null;
return { items, nextCursor: next, changeToken: `token-${library.length}` };
}),
asset: vi.fn(),
changes: vi.fn(),
exportOriginal: vi.fn(async (id: string, destination: string, opts) => {
active += 1;
stats.maxActiveExports = Math.max(stats.maxActiveExports, active);
try {
await new Promise((resolve) => setTimeout(resolve, 5));
opts?.signal?.throwIfAborted();
const contents = options.contents?.(id) ?? `original bytes of ${id}`;
const file = path.join(destination, "IMG.MOV");
await writeFile(file, contents);
exports.push(destination);
return {
path: file,
fileName: "IMG.MOV",
sizeBytes: Buffer.byteLength(contents),
sha256: sha(contents),
};
} finally {
active -= 1;
}
}),
};
return { helper, exports, stats };
}
function memoryStore(initial?: PhotosSyncState) {
let state: PhotosSyncState = initial ?? { version: 1, deviceId: "device-1", targets: {} };
const writes: PhotosSyncState[] = [];
const store: PhotosSyncStore = {
read: async () => JSON.parse(JSON.stringify(state)) as PhotosSyncState,
write: async (next) => {
state = JSON.parse(JSON.stringify(next)) as PhotosSyncState;
writes.push(state);
},
};
return { store, writes, current: () => state };
}
function asset(id: string, externalId: string, ingestState: MediaAsset["ingestState"]) {
return {
id,
sourceId: "source-1",
externalId,
kind: "video",
storage: "remote",
capturedAt: null,
durationMs: null,
width: null,
height: null,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
ingestState,
hasProxy: false,
hasThumbnail: false,
qualityScore: null,
createdAt: "2024-01-01T00:00:00.000Z",
} as MediaAsset;
}
function fakeApi(
options: {
assets?: MediaAsset[];
chunkBytes?: number;
uploads?: Record<string, MediaUpload>;
} = {},
) {
const uploads = new Map(Object.entries(options.uploads ?? {}));
const pushes: {
items: number;
syncToken: string | null;
complete: boolean;
deleted?: number;
}[] = [];
const chunkBytes = options.chunkBytes ?? 4;
const api = {
media: {
sources: {
registerClient: vi.fn(async (input: { deviceId: string }) => ({
id: "source-1",
kind: "apple_photos" as const,
label: "Apple Photos",
status: "idle" as const,
assetCount: 0,
lastSyncedAt: null,
error: null,
createdAt: "2024-01-01T00:00:00.000Z",
deviceId: input.deviceId,
})),
pushItems: vi.fn(
async (input: {
items: unknown[];
deletedExternalIds?: string[];
syncToken: string | null;
complete: boolean;
}) => {
const deleted = input.deletedExternalIds?.length ?? 0;
expect(input.items.length > 0 || deleted > 0 || input.complete).toBe(true);
expect(input.items.length).toBeLessThanOrEqual(500);
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
...(deleted > 0 ? { deleted } : {}),
});
return { created: input.items.length, updated: 0, unchanged: 0, deleted };
},
),
},
assets: {
search: vi.fn(async (input: { cursor?: string | null }) => {
const all = options.assets ?? [];
// Two pages to exercise pagination.
const half = Math.ceil(all.length / 2);
return input.cursor === "p2"
? { items: all.slice(half), nextCursor: null }
: { items: all.slice(0, half), nextCursor: all.length > 1 ? "p2" : null };
}),
},
uploads: {
begin: vi.fn(async (input: { assetId: string; sizeBytes: number }) => {
const upload: MediaUpload = {
id: `upload-${input.assetId}`,
assetId: input.assetId,
sizeBytes: input.sizeBytes,
chunkBytes,
receivedChunks: [],
status: "open",
};
uploads.set(upload.id, upload);
return upload;
}),
status: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId);
if (!upload) throw Object.assign(new Error("Not found"), { code: "NOT_FOUND" });
return upload;
}),
complete: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId)!;
return { ...upload, status: "complete" as const };
}),
},
},
};
return { api: api as unknown as MediaSyncApi & typeof api, pushes };
}
let tempRoot: string;
beforeEach(async () => {
tempRoot = await mkdtemp(path.join(tmpdir(), "rakazo-photos-sync-"));
});
afterEach(async () => {
await rm(tempRoot, { recursive: true, force: true });
});
const baseOptions = () => ({
target: "https://rakazo.example.com#space_1",
label: "Apple Photos",
tempRoot,
signal: new AbortController().signal,
});
describe("Apple Photos metadata sync", () => {
it("registers once, pushes batches of at most 500 and completes the listing", async () => {
const { helper } = fakeHelper({ size: 1203 });
const { api, pushes } = fakeApi();
const { store, current } = memoryStore();
const chunks = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledWith(
{ kind: "apple_photos", label: "Apple Photos", deviceId: "device-1" },
expect.anything(),
);
expect(pushes).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 500, syncToken: null, complete: false },
{ items: 203, syncToken: "token-1203", complete: true },
]);
expect(result).toMatchObject({ indexed: 1203, created: 1203, uploaded: 0, fullListing: true });
expect(current().targets["https://rakazo.example.com#space_1"]).toEqual({
sourceId: "source-1",
listing: null,
changeToken: "token-1203",
uploads: {},
});
// The next run reads changes only and does not register again.
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "token-next",
changed: [libraryItem(1), libraryItem(2)],
deleted: [],
fullResyncRequired: false,
});
const second = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(helper.changes).toHaveBeenCalledWith("token-1203", expect.anything());
expect(pushes.at(-1)).toEqual({ items: 2, syncToken: "token-next", complete: false });
expect(second.fullListing).toBe(false);
expect(current().targets["https://rakazo.example.com#space_1"]?.changeToken).toBe("token-next");
});
it("persists the cursor after every batch and resumes an interrupted listing", async () => {
const { helper } = fakeHelper({ size: 1100 });
const { api, pushes } = fakeApi();
const { store, writes, current } = memoryStore();
vi.mocked(api.media.sources.pushItems).mockImplementationOnce(async (input) => {
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
});
return { created: input.items.length, updated: 0, unchanged: 0 };
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(new Error("network down"));
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() }),
).rejects.toThrow("network down");
const interrupted = current().targets["https://rakazo.example.com#space_1"]!;
expect(interrupted.listing).toMatchObject({ cursor: "500", token: "token-1100" });
expect(interrupted.listing?.lastItem?.externalId).toBe(libraryItem(499).externalId);
expect(writes.length).toBeGreaterThan(1);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(vi.mocked(helper.list).mock.calls.at(-2)?.[0]).toMatchObject({ cursor: "500" });
expect(pushes.slice(1)).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 100, syncToken: "token-1100", complete: true },
]);
expect(current().targets["https://rakazo.example.com#space_1"]).toMatchObject({
listing: null,
changeToken: "token-1100",
});
});
it("closes a resumed listing whose last page came back empty", async () => {
const { helper } = fakeHelper({ size: 0 });
const { api, pushes } = fakeApi();
const last = libraryItem(7);
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: { cursor: "500", token: "token-a", lastItem: last },
changeToken: null,
uploads: {},
},
},
});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(pushes).toEqual([{ items: 0, syncToken: "token-a", complete: true }]);
});
it("pushes device deletions without relisting", async () => {
const { helper } = fakeHelper({ size: 3 });
const { api, pushes } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "old",
uploads: {},
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "new",
changed: [],
deleted: ["gone/L0/001"],
fullResyncRequired: false,
});
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: vi.fn(),
});
expect(result.fullListing).toBe(false);
expect(pushes).toEqual([{ items: 0, syncToken: "new", complete: false, deleted: 1 }]);
});
it("registers again when the server no longer knows the source", async () => {
const { helper } = fakeHelper({ size: 2 });
const { api } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "deleted-source",
listing: { cursor: null, token: null, lastItem: null },
changeToken: null,
uploads: {},
},
},
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(
Object.assign(new Error("Not found"), { code: "NOT_FOUND" }),
);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(vi.mocked(api.media.sources.pushItems).mock.calls.at(-1)?.[0].sourceId).toBe("source-1");
});
it("stops before anything is sent when Photos access is denied", async () => {
const { helper } = fakeHelper({ size: 2 });
vi.mocked(helper.authorize).mockResolvedValueOnce("denied");
const { api } = fakeApi();
await expect(
syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk: vi.fn(),
}),
).rejects.toMatchObject({ code: "not_authorized" });
expect(api.media.sources.registerClient).not.toHaveBeenCalled();
});
});
describe("Apple Photos original upload", () => {
const awaitingAssets = [
asset("a1", "ASSET-00001/L0/001", "awaiting_upload"),
asset("a2", "ASSET-00002/L0/001", "proxied"),
asset("a3", "ASSET-00003/L0/001", "awaiting_upload"),
asset("a4", "ASSET-00004/L0/001", "awaiting_upload"),
];
it("uploads every awaiting original in chunks with two at a time and cleans up", async () => {
const { helper, exports, stats } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: awaitingAssets, chunkBytes: 8 });
const sent = new Map<string, Buffer[]>();
const uploadChunk = vi.fn(async (uploadId: string, index: number, bytes: Uint8Array) => {
const parts = sent.get(uploadId) ?? [];
parts[index] = Buffer.from(bytes);
sent.set(uploadId, parts);
});
const progress = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk,
onProgress: progress,
});
expect(result.uploaded).toBe(3);
expect(
vi
.mocked(helper.exportOriginal)
.mock.calls.map((call) => call[0])
.sort(),
).toEqual(["ASSET-00001/L0/001", "ASSET-00003/L0/001", "ASSET-00004/L0/001"]);
expect(stats.maxActiveExports).toBe(2);
expect(Buffer.concat(sent.get("upload-a1")!).toString()).toBe(
"original bytes of ASSET-00001/L0/001",
);
expect(sent.get("upload-a1")).toHaveLength(Math.ceil(36 / 8));
expect(api.media.uploads.begin).toHaveBeenCalledWith(
expect.objectContaining({
assetId: "a1",
sizeBytes: 36,
sha256: sha("original bytes of ASSET-00001/L0/001"),
}),
expect.anything(),
);
expect(api.media.uploads.complete).toHaveBeenCalledTimes(3);
expect(progress).toHaveBeenLastCalledWith({
phase: "uploading",
indexed: 1,
uploadsDone: 3,
uploadsTotal: 3,
});
for (const dir of exports) await expect(readdir(dir)).rejects.toThrow();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("resumes an open upload and sends only the missing chunks", async () => {
const { helper } = fakeHelper({ size: 1 });
const contents = "original bytes of ASSET-00001/L0/001";
const { api } = fakeApi({
assets: [awaitingAssets[0]!],
uploads: {
"upload-old": {
id: "upload-old",
assetId: "a1",
sizeBytes: Buffer.byteLength(contents),
chunkBytes: 10,
receivedChunks: [0, 2],
status: "open",
},
},
});
const { store, current } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "tok",
uploads: { a1: "upload-old" },
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "tok",
changed: [],
deleted: [],
fullResyncRequired: false,
});
const uploadChunk = vi.fn(async () => {});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(
uploadChunk.mock.calls.map((call) => [
call[0],
call[1],
Buffer.from(call[2] as Uint8Array).toString(),
]),
).toEqual([
["upload-old", 1, contents.slice(10, 20)],
["upload-old", 3, contents.slice(30)],
]);
expect(current().targets["https://rakazo.example.com#space_1"]?.uploads).toEqual({});
});
it("aborts on a checksum mismatch without starting an upload", async () => {
const { helper } = fakeHelper({ size: 1 });
vi.mocked(helper.exportOriginal).mockImplementationOnce(async (_id, destination) => {
const file = path.join(destination, "IMG.MOV");
await writeFile(file, "tampered");
return { path: file, fileName: "IMG.MOV", sizeBytes: 8, sha256: sha("original") };
});
const { api } = fakeApi({ assets: [awaitingAssets[0]!] });
const uploadChunk = vi.fn();
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store: memoryStore().store, uploadChunk }),
).rejects.toMatchObject({ code: "integrity" });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(uploadChunk).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("cancels a running upload", async () => {
const { helper } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: [awaitingAssets[0]!], chunkBytes: 4 });
const controller = new AbortController();
const uploadChunk = vi.fn(
async (_id: string, index: number, _bytes: Uint8Array, signal: AbortSignal) => {
if (index === 1) controller.abort();
signal.throwIfAborted();
},
);
await expect(
syncApplePhotos({
...baseOptions(),
signal: controller.signal,
helper,
api,
store: memoryStore().store,
uploadChunk,
}),
).rejects.toMatchObject({ code: "cancelled" });
expect(uploadChunk).toHaveBeenCalledTimes(2);
expect(api.media.uploads.complete).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
});
describe("Apple Photos sync state file", () => {
it("keeps a stable device id across launches", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-state-"));
try {
const store = createPhotosSyncFileStore(dir, () => "device-a");
const first = await store.read();
expect(first.deviceId).toBe("device-a");
first.targets.x = { sourceId: "s", listing: null, changeToken: "t", uploads: {} };
await store.write(first);
const reopened = await createPhotosSyncFileStore(dir, () => "device-b").read();
expect(reopened).toEqual(first);
expect(await readFile(path.join(dir, "media-photos-sync.json"), "utf8")).toContain(
"device-a",
);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("drops unreadable state and malformed targets", () => {
expect(parsePhotosSyncState("{")).toBeNull();
expect(
parsePhotosSyncState(JSON.stringify({ version: 2, deviceId: "d", targets: {} })),
).toBeNull();
expect(
parsePhotosSyncState(
JSON.stringify({
version: 1,
deviceId: "d",
targets: {
good: { sourceId: "s", listing: null, changeToken: null, uploads: {} },
bad: { sourceId: 4, listing: null, changeToken: null, uploads: {} },
badListing: { sourceId: "s", listing: { cursor: 3 }, changeToken: null, uploads: {} },
},
}),
)?.targets,
).toEqual({ good: { sourceId: "s", listing: null, changeToken: null, uploads: {} } });
});
});
describe("chunk uploader", () => {
it("PUTs raw bytes with the session and space", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 204 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: "space_1",
fetch,
});
const signal = new AbortController().signal;
await upload("up/1", 3, new Uint8Array([1, 2, 3]), signal);
expect(fetch).toHaveBeenCalledWith(
"https://rakazo.example.com/api/media/uploads/up%2F1/chunks/3",
expect.objectContaining({
method: "PUT",
credentials: "include",
redirect: "error",
signal,
headers: { "content-type": "application/octet-stream", "x-rakazo-space-id": "space_1" },
}),
);
});
it("rejects failed chunks and non-origin targets", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 409 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: null,
fetch,
});
await expect(
upload("u", 0, new Uint8Array(), new AbortController().signal),
).rejects.toMatchObject({
code: "upload_failed",
});
expect(() =>
createChunkUploader({ origin: "https://rakazo.example.com/rpc", spaceId: null, fetch }),
).toThrow();
expect(() => createChunkUploader({ origin: "file:///etc", spaceId: null, fetch })).toThrow();
});
});
describe("runPool", () => {
it("bounds concurrency and reports the first failure", async () => {
let active = 0;
let max = 0;
const seen: number[] = [];
await expect(
runPool([1, 2, 3, 4, 5], 2, new AbortController().signal, async (item, signal) => {
active += 1;
max = Math.max(max, active);
await new Promise((resolve) => setTimeout(resolve, 5));
active -= 1;
if (item === 2) throw new Error("boom");
signal.throwIfAborted();
seen.push(item);
}),
).rejects.toThrow("boom");
expect(max).toBe(2);
expect(seen).not.toContain(4);
});
});

View File

@ -0,0 +1,581 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { mkdtemp, open, rm, stat } from "node:fs/promises";
import path from "node:path";
import type {
ClientMediaItem,
MediaAsset,
MediaUpload,
mediaRpcShapes,
PhotosSyncProgress,
PhotosSyncResult,
} from "@rakazo/contracts";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PHOTOS_MAX_PAGE } from "./media-photos-helper.js";
import { readPrivateFile, writePrivateFile } from "./setup-store.js";
/*
* Apple Photos is a client-pushed source: the API may run on another machine, so the
* app never sends a local path. It pushes metadata in batches and uploads the originals
* the server asks for (`awaiting_upload`) in resumable raw-byte chunks.
*/
type Shapes = typeof mediaRpcShapes;
type Shape = { input: { _input: unknown }; output: { _output: unknown } };
/** One oRPC procedure as `ContractRouterClient<AppContract>` exposes it. */
type Procedure<S extends Shape> = (
input: S["input"]["_input"],
options?: { signal?: AbortSignal },
) => Promise<S["output"]["_output"]>;
/** The slice of the app RPC client the sync needs; `rpc` from a contract client satisfies it. */
export interface MediaSyncApi {
media: {
sources: {
registerClient: Procedure<Shapes["sources"]["registerClient"]>;
pushItems: Procedure<Shapes["sources"]["pushItems"]>;
};
assets: { search: Procedure<Shapes["assets"]["search"]> };
uploads: {
begin: Procedure<Shapes["uploads"]["begin"]>;
status: Procedure<Shapes["uploads"]["status"]>;
complete: Procedure<Shapes["uploads"]["complete"]>;
};
};
}
/** Sends one raw chunk to `PUT /api/media/uploads/:uploadId/chunks/:index`. */
export type UploadChunk = (
uploadId: string,
index: number,
bytes: Uint8Array<ArrayBuffer>,
signal: AbortSignal,
) => Promise<void>;
export type PhotosSyncErrorCode = "not_authorized" | "cancelled" | "integrity" | "upload_failed";
export class PhotosSyncError extends Error {
constructor(
readonly code: PhotosSyncErrorCode,
message: string,
) {
super(message);
this.name = "PhotosSyncError";
}
}
// ---------------------------------------------------------------------------
// Persistent state
// ---------------------------------------------------------------------------
export interface PhotosSyncTargetState {
sourceId: string;
/** A full listing in progress: where to resume, the token from its start and the last pushed item. */
listing: { cursor: string | null; token: string | null; lastItem: ClientMediaItem | null } | null;
/** Set after a complete listing; later syncs only read changes since it. */
changeToken: string | null;
/** Open uploads by asset id so an interrupted upload resumes instead of restarting. */
uploads: Record<string, string>;
}
export interface PhotosSyncState {
version: 1;
/** Stable per installation; the server keys the Apple Photos source by it. */
deviceId: string;
/** Keyed by server origin and space, since each has its own source. */
targets: Record<string, PhotosSyncTargetState>;
}
export interface PhotosSyncStore {
read(): Promise<PhotosSyncState>;
write(state: PhotosSyncState): Promise<void>;
}
export const PHOTOS_SYNC_FILE_NAME = "media-photos-sync.json";
const MAX_STATE_BYTES = 4 * 1024 * 1024;
/** Kept next to the desktop setup file, private to the user and written atomically. */
export function createPhotosSyncFileStore(
userDataDir: string,
newDeviceId: () => string = randomUUID,
): PhotosSyncStore {
const file = path.join(userDataDir, PHOTOS_SYNC_FILE_NAME);
return {
async read() {
const raw = await readPrivateFile(file, MAX_STATE_BYTES);
return (raw === null ? null : parsePhotosSyncState(raw)) ?? emptyState(newDeviceId());
},
write: (state) => writePrivateFile(file, `${JSON.stringify(state)}\n`),
};
}
function emptyState(deviceId: string): PhotosSyncState {
return { version: 1, deviceId, targets: {} };
}
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" && value !== null && !Array.isArray(value);
const isNullableString = (value: unknown) => value === null || typeof value === "string";
/** Unreadable state starts over: the server deduplicates by device and external id. */
export function parsePhotosSyncState(raw: string): PhotosSyncState | null {
let value: unknown;
try {
value = JSON.parse(raw);
} catch {
return null;
}
if (
!isRecord(value) ||
value.version !== 1 ||
typeof value.deviceId !== "string" ||
value.deviceId === "" ||
value.deviceId.length > 200 ||
!isRecord(value.targets)
) {
return null;
}
const targets: Record<string, PhotosSyncTargetState> = {};
for (const [key, target] of Object.entries(value.targets)) {
if (
!isRecord(target) ||
typeof target.sourceId !== "string" ||
!isNullableString(target.changeToken) ||
!isRecord(target.uploads) ||
!Object.values(target.uploads).every((id) => typeof id === "string")
) {
continue;
}
const listing = target.listing;
const validListing =
listing === null ||
(isRecord(listing) &&
isNullableString(listing.cursor) &&
isNullableString(listing.token) &&
(listing.lastItem === null || isRecord(listing.lastItem)));
if (!validListing) continue;
targets[key] = target as unknown as PhotosSyncTargetState;
}
return { version: 1, deviceId: value.deviceId, targets };
}
// ---------------------------------------------------------------------------
// Chunk upload over HTTP
// ---------------------------------------------------------------------------
/**
* Chunk PUTs ride the same signed-in session as the app window: the injected `fetch`
* must carry that session's cookies (Electron `session.fetch` with credentials).
*/
export function createChunkUploader(options: {
origin: string;
spaceId: string | null;
fetch: (url: string, init: RequestInit) => Promise<Response>;
}): UploadChunk {
const origin = new URL(options.origin);
if (!["http:", "https:"].includes(origin.protocol) || origin.origin !== options.origin) {
throw new Error("Uploads need the server origin.");
}
return async (uploadId, index, bytes, signal) => {
if (!Number.isSafeInteger(index) || index < 0) throw new Error("Invalid chunk index.");
const headers: Record<string, string> = { "content-type": "application/octet-stream" };
if (options.spaceId) headers["x-rakazo-space-id"] = options.spaceId;
const response = await options.fetch(
`${origin.origin}/api/media/uploads/${encodeURIComponent(uploadId)}/chunks/${index}`,
{
method: "PUT",
headers,
body: bytes,
credentials: "include",
redirect: "error",
signal,
},
);
if (!response.ok) {
throw new PhotosSyncError(
"upload_failed",
`The server rejected an upload (${response.status}).`,
);
}
};
}
// ---------------------------------------------------------------------------
// Sync
// ---------------------------------------------------------------------------
export type { PhotosSyncProgress, PhotosSyncResult };
export interface PhotosSyncOptions {
helper: PhotosHelper;
api: MediaSyncApi;
uploadChunk: UploadChunk;
store: PhotosSyncStore;
/** Server origin plus space, e.g. `https://rakazo.example.com#space_1`. */
target: string;
/** Source label shown in the library, e.g. the Mac's name or "Apple Photos". */
label: string;
/** Exports land in a fresh folder below this and are deleted after upload. */
tempRoot: string;
signal: AbortSignal;
onProgress?: (progress: PhotosSyncProgress) => void;
/** At most 500, the contract's batch limit. */
batchSize?: number;
concurrency?: number;
}
const MAX_SYNC_TOKEN_LENGTH = 2000;
export async function syncApplePhotos(options: PhotosSyncOptions): Promise<PhotosSyncResult> {
const { helper, api, signal } = options;
const batchSize = Math.min(Math.max(1, options.batchSize ?? PHOTOS_MAX_PAGE), PHOTOS_MAX_PAGE);
const result: PhotosSyncResult = {
indexed: 0,
created: 0,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: false,
};
const progress = { uploadsDone: 0, uploadsTotal: 0 };
const report = (phase: PhotosSyncProgress["phase"]) =>
options.onProgress?.({ phase, indexed: result.indexed, ...progress });
const state = await options.store.read();
// Writes are serialized so concurrent upload workers never interleave partial state.
let saving = Promise.resolve();
const save = () => {
const snapshot = JSON.parse(JSON.stringify(state)) as PhotosSyncState;
saving = saving.then(() => options.store.write(snapshot));
return saving;
};
try {
report("authorizing");
const authorization = await helper.authorize(true, signal);
if (authorization !== "authorized" && authorization !== "limited") {
throw new PhotosSyncError("not_authorized", "Photos access was not granted.");
}
const ensureTarget = async (): Promise<PhotosSyncTargetState> => {
const existing = state.targets[options.target];
if (existing) return existing;
const source = await api.media.sources.registerClient(
{ kind: "apple_photos", label: options.label, deviceId: state.deviceId },
{ signal },
);
const created: PhotosSyncTargetState = {
sourceId: source.id,
listing: null,
changeToken: null,
uploads: {},
};
state.targets[options.target] = created;
await save();
return created;
};
const push = async (
target: PhotosSyncTargetState,
items: ClientMediaItem[],
syncToken: string | null,
complete: boolean,
deletedExternalIds: string[] = [],
) => {
const counts = await api.media.sources.pushItems(
{
sourceId: target.sourceId,
items,
deletedExternalIds,
syncToken:
syncToken !== null && syncToken.length <= MAX_SYNC_TOKEN_LENGTH ? syncToken : null,
complete,
},
{ signal },
);
result.indexed += items.length;
result.created += counts.created;
result.updated += counts.updated;
result.unchanged += counts.unchanged;
report("indexing");
};
/** Full listing, resumable page by page; the last push says `complete`. */
const listAll = async (target: PhotosSyncTargetState) => {
result.fullListing = true;
target.listing ??= { cursor: null, token: null, lastItem: null };
const listing = target.listing;
while (true) {
const page = await helper.list({ cursor: listing.cursor, limit: batchSize }, signal);
if (listing.cursor === null) listing.token = page.changeToken;
const done = page.nextCursor === null;
const items = page.items;
// An empty final page still closes the listing.
if (items.length > 0 || done) await push(target, items, done ? listing.token : null, done);
if (done) {
target.changeToken = listing.token;
target.listing = null;
await save();
return;
}
listing.cursor = page.nextCursor;
listing.lastItem = items.at(-1) ?? listing.lastItem;
await save();
}
};
const syncMetadata = async () => {
const target = await ensureTarget();
if (target.listing !== null || target.changeToken === null) {
await listAll(target);
return;
}
const changes = await helper.changes(target.changeToken, signal);
if (changes.fullResyncRequired || changes.token === null) {
target.listing = null;
await listAll(target);
return;
}
const batches = Math.max(
Math.ceil(changes.changed.length / batchSize),
Math.ceil(changes.deleted.length / DELETED_BATCH_SIZE),
);
for (let index = 0; index < batches; index++) {
const batch = changes.changed.slice(index * batchSize, (index + 1) * batchSize);
const deleted = changes.deleted.slice(
index * DELETED_BATCH_SIZE,
(index + 1) * DELETED_BATCH_SIZE,
);
const last = index === batches - 1;
await push(target, batch, last ? changes.token : null, false, deleted);
}
target.changeToken = changes.token;
await save();
};
report("indexing");
try {
await syncMetadata();
} catch (error) {
// The source was removed on the server: register again and list from scratch.
if (!isNotFound(error) || signal.aborted) throw error;
delete state.targets[options.target];
await save();
await syncMetadata();
}
const target = state.targets[options.target]!;
const awaiting = await awaitingUploads(api, target.sourceId, signal);
progress.uploadsTotal = awaiting.length;
// Forget uploads for assets the server no longer waits for.
const waiting = new Set(awaiting.map((asset) => asset.id));
for (const assetId of Object.keys(target.uploads)) {
if (!waiting.has(assetId)) delete target.uploads[assetId];
}
await save();
report("uploading");
await runPool(awaiting, options.concurrency ?? 2, signal, async (asset, workerSignal) => {
await uploadAsset({ ...options, signal: workerSignal }, target, asset, save);
result.uploaded += 1;
progress.uploadsDone += 1;
report("uploading");
});
return result;
} catch (error) {
if (signal.aborted) throw new PhotosSyncError("cancelled", "Sync cancelled.");
throw error;
} finally {
await saving.catch(() => undefined);
}
}
/**
* Binds the sync to the server and space the request names. `connect` builds the RPC
* client and chunk uploader on the signed-in session for that space.
*/
export function createPhotosSyncRunner(deps: {
helper: PhotosHelper;
store: PhotosSyncStore;
tempRoot: string;
label: string;
connect: (
spaceId: string | null,
) => Promise<{ origin: string; api: MediaSyncApi; uploadChunk: UploadChunk }>;
}) {
return async (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}): Promise<PhotosSyncResult> => {
const { origin, api, uploadChunk } = await deps.connect(request.spaceId);
return syncApplePhotos({
helper: deps.helper,
store: deps.store,
tempRoot: deps.tempRoot,
label: deps.label,
api,
uploadChunk,
target: `${origin}#${request.spaceId ?? "default"}`,
signal: request.signal,
onProgress: request.onProgress,
});
};
}
function isNotFound(error: unknown): boolean {
return isRecord(error) && error.code === "NOT_FOUND";
}
interface AwaitingAsset {
id: string;
externalId: string;
}
/** Assets of this source the server is waiting for, mapped back to Photos items. */
const DELETED_BATCH_SIZE = 5000;
export async function awaitingUploads(
api: MediaSyncApi,
sourceId: string,
signal: AbortSignal,
): Promise<AwaitingAsset[]> {
const awaiting: AwaitingAsset[] = [];
let cursor: string | null = null;
do {
const page: { items: MediaAsset[]; nextCursor: string | null } = await api.media.assets.search(
{ sourceId, ingestStates: ["awaiting_upload"], cursor, limit: PHOTOS_MAX_PAGE },
{ signal },
);
for (const asset of page.items) {
if (asset.ingestState !== "awaiting_upload" || asset.sourceId !== sourceId) continue;
awaiting.push({ id: asset.id, externalId: asset.externalId });
}
cursor = page.nextCursor;
} while (cursor !== null);
return awaiting;
}
async function uploadAsset(
options: PhotosSyncOptions,
target: PhotosSyncTargetState,
asset: AwaitingAsset,
save: () => Promise<void>,
): Promise<void> {
const { api, signal } = options;
const directory = await mkdtemp(path.join(options.tempRoot, "rakazo-photos-"));
try {
const exported = await options.helper.exportOriginal(asset.externalId, directory, { signal });
// The helper's hash is checked against the bytes on disk before anything is sent.
const local = await hashFile(exported.path, signal);
if (local.sha256 !== exported.sha256 || local.sizeBytes !== exported.sizeBytes) {
throw new PhotosSyncError("integrity", "The exported file does not match its checksum.");
}
let upload: MediaUpload | null = null;
const previous = target.uploads[asset.id];
if (previous) {
const status = await api.media.uploads
.status({ uploadId: previous }, { signal })
.catch((error: unknown) => {
if (isNotFound(error)) return null;
throw error;
});
if (status?.status === "complete") {
delete target.uploads[asset.id];
await save();
return;
}
if (status?.status === "open" && status.sizeBytes === local.sizeBytes) upload = status;
}
if (upload === null) {
upload = await api.media.uploads.begin(
{
assetId: asset.id,
sizeBytes: local.sizeBytes,
sha256: local.sha256,
fileName: exported.fileName.slice(0, 255),
},
{ signal },
);
target.uploads[asset.id] = upload.id;
await save();
}
const received = new Set(upload.receivedChunks);
const chunkCount = Math.ceil(local.sizeBytes / upload.chunkBytes);
const file = await open(exported.path, "r");
try {
for (let index = 0; index < chunkCount; index += 1) {
if (received.has(index)) continue;
signal.throwIfAborted();
const offset = index * upload.chunkBytes;
const length = Math.min(upload.chunkBytes, local.sizeBytes - offset);
const bytes = Buffer.alloc(length);
const { bytesRead } = await file.read(bytes, 0, length, offset);
if (bytesRead !== length) {
throw new PhotosSyncError("integrity", "The exported file changed during upload.");
}
await options.uploadChunk(upload.id, index, bytes, signal);
}
} finally {
await file.close();
}
const completed = await api.media.uploads.complete({ uploadId: upload.id }, { signal });
if (completed.status !== "complete") {
throw new PhotosSyncError("upload_failed", "The server did not accept the upload.");
}
delete target.uploads[asset.id];
await save();
} finally {
await rm(directory, { recursive: true, force: true });
}
}
export async function hashFile(
file: string,
signal?: AbortSignal,
): Promise<{ sha256: string; sizeBytes: number }> {
const hash = createHash("sha256");
for await (const chunk of createReadStream(file, { signal })) hash.update(chunk as Buffer);
return { sha256: hash.digest("hex"), sizeBytes: (await stat(file)).size };
}
/**
* Runs `work` with at most `limit` items in flight. The first failure aborts the
* others and is the error reported.
*/
export async function runPool<T>(
items: T[],
limit: number,
signal: AbortSignal,
work: (item: T, signal: AbortSignal) => Promise<void>,
): Promise<void> {
const controller = new AbortController();
const forward = () => controller.abort(signal.reason);
signal.addEventListener("abort", forward, { once: true });
if (signal.aborted) forward();
let next = 0;
let failed = false;
let failure: unknown;
const worker = async () => {
while (!failed && !controller.signal.aborted && next < items.length) {
const item = items[next++]!;
try {
await work(item, controller.signal);
} catch (error) {
if (!failed) {
failed = true;
failure = error;
controller.abort(error);
}
}
}
};
try {
await Promise.all(Array.from({ length: Math.min(Math.max(1, limit), items.length) }, worker));
} finally {
signal.removeEventListener("abort", forward);
}
if (failed) throw failure;
signal.throwIfAborted();
}

View File

@ -9,6 +9,18 @@ contextBridge.exposeInMainWorld("rakazoDesktop", {
request: (pathname, body) =>
ipcRenderer.invoke("desktop.localSettings.request", pathname, body),
},
media: {
photos: {
status: () => ipcRenderer.invoke("desktop.media.photos.status"),
sync: (request) => ipcRenderer.invoke("desktop.media.photos.sync", request),
cancel: () => ipcRenderer.invoke("desktop.media.photos.cancel"),
onProgress: (listener) => {
const handler = (_event, progress) => listener(progress);
ipcRenderer.on("desktop.media.photos.progress", handler);
return () => ipcRenderer.off("desktop.media.photos.progress", handler);
},
},
},
window: {
close: () => ipcRenderer.invoke("desktop.window.close"),
minimize: () => ipcRenderer.invoke("desktop.window.minimize"),
@ -21,6 +33,22 @@ contextBridge.exposeInMainWorld("rakazoDesktop", {
download: () => ipcRenderer.invoke("desktop.update.download"),
install: () => ipcRenderer.invoke("desktop.update.install"),
},
browserPreview: {
state: () => ipcRenderer.invoke("desktop.browserPreview.state"),
show: (bounds) => ipcRenderer.invoke("desktop.browserPreview.show", bounds),
hide: () => ipcRenderer.invoke("desktop.browserPreview.hide"),
navigate: (url) => ipcRenderer.invoke("desktop.browserPreview.navigate", url),
back: () => ipcRenderer.invoke("desktop.browserPreview.back"),
forward: () => ipcRenderer.invoke("desktop.browserPreview.forward"),
reload: () => ipcRenderer.invoke("desktop.browserPreview.reload"),
stop: () => ipcRenderer.invoke("desktop.browserPreview.stop"),
openDevTools: () => ipcRenderer.invoke("desktop.browserPreview.openDevTools"),
onState: (listener) => {
const handler = (_event, state) => listener(state);
ipcRenderer.on("desktop.browserPreview.state", handler);
return () => ipcRenderer.off("desktop.browserPreview.state", handler);
},
},
oauth: {
open: (url) => ipcRenderer.invoke("desktop.oauth.open", url),
cancel: (url) => ipcRenderer.invoke("desktop.oauth.cancel", url),

View File

@ -32,8 +32,10 @@ describe("desktop preload bridge", () => {
expect(globalName).toBe("rakazoDesktop");
expect(bridge.platform).toBe("linux");
expect(Object.keys(bridge).sort()).toEqual([
"browserPreview",
"files",
"localSettings",
"media",
"oauth",
"platform",
"update",
@ -46,6 +48,24 @@ describe("desktop preload bridge", () => {
"toggleMaximize",
]);
expect(Object.keys(bridge.update).sort()).toEqual(["check", "download", "install", "state"]);
expect(Object.keys(bridge.media?.photos ?? {}).sort()).toEqual([
"cancel",
"onProgress",
"status",
"sync",
]);
expect(Object.keys(bridge.browserPreview ?? {}).sort()).toEqual([
"back",
"forward",
"hide",
"navigate",
"onState",
"openDevTools",
"reload",
"show",
"state",
"stop",
]);
await bridge.files?.pickDirectory();
await bridge.oauth.open?.("https://provider.example.com/authorize");
@ -58,6 +78,8 @@ describe("desktop preload bridge", () => {
await bridge.update.check();
await bridge.update.download();
await bridge.update.install();
await bridge.browserPreview?.navigate("http://localhost:5173/");
await bridge.browserPreview?.show({ x: 1, y: 2, width: 3, height: 4 });
expect(invoke.mock.calls.map(([channel]) => channel)).toEqual([
"desktop.files.pickDirectory",
"desktop.oauth.open",
@ -70,6 +92,8 @@ describe("desktop preload bridge", () => {
"desktop.update.check",
"desktop.update.download",
"desktop.update.install",
"desktop.browserPreview.navigate",
"desktop.browserPreview.show",
]);
});
@ -77,8 +101,10 @@ describe("desktop preload bridge", () => {
const { exposeInMainWorld } = runPreload("preload.cjs");
const [, bridge] = exposeInMainWorld.mock.calls[0] as [string, Record<string, unknown>];
expect(Object.keys(bridge).sort()).toEqual([
"browserPreview",
"files",
"localSettings",
"media",
"oauth",
"platform",
"update",

View File

@ -1074,7 +1074,7 @@ export function applyMobileThreadEvent(
messages: [...remaining, streaming],
};
}
if (event.type === "agent.tool.completed") {
if (event.type === "agent.tool.completed" || event.type === "thread.computer") {
return { ...prev, cursor: event.seq ?? prev.cursor };
}
if (event.type === "thread.subagent") {

View File

@ -35,6 +35,7 @@
"expo-image-picker": "57.0.20",
"expo-linking": "57.0.11",
"expo-localization": "57.0.2",
"expo-media-library": "57.0.5",
"expo-modules-core": "57.0.20",
"expo-notifications": "57.0.21",
"expo-router": "57.0.24",
@ -44,6 +45,7 @@
"expo-status-bar": "57.0.1",
"expo-symbols": "57.0.3",
"expo-updates": "57.0.24",
"expo-video": "57.0.5",
"react": "19.2.3",
"react-dom": "19.2.3",
"react-native": "0.86.3",

View File

@ -0,0 +1,53 @@
import { expect, test } from "@playwright/test";
import { activeBotId, captureScreenshot, completeOnboarding, rpc, signup } from "./helpers";
const SCREEN_URL = "https://screen.example/embed.html";
test("the bot's cursor glides to where it clicks on its live screen", async ({
page,
}, testInfo) => {
await signup(page, `agent-cursor-${Date.now()}@rakazo.test`, "password12", "Cursor Watcher");
await completeOnboarding(page);
const botId = activeBotId(page);
await rpc(page, "computer/boot", { botId });
// Stand in for the noVNC viewer: a 16:10 desktop with a field in the middle.
await page.route("https://screen.example/**", (route) =>
route.fulfill({
contentType: "text/html",
body: `<!doctype html><title>Desktop</title><body style="margin:0;background:#1d1d20;display:grid;place-items:center;height:100vh">
<input style="width:40%;padding:8px;font:16px system-ui" placeholder="Search">`,
}),
);
await page.route("**/rpc/computer/screenUrl", (route) =>
route.fulfill({
contentType: "application/json",
body: JSON.stringify({ json: { url: SCREEN_URL } }),
}),
);
await page.getByTitle("Agent computer").click();
const preview = page.getByTestId("computer-preview");
await expect(preview.locator("iframe")).toHaveAttribute("src", SCREEN_URL);
const cursor = preview.getByTestId("agent-cursor");
await expect(cursor).toHaveCSS("opacity", "0");
// The scripted runtime clicks the screen center (640, 400) and types.
await rpc(page, "threads/send", { botId, text: "use your screen and type cursor-demo" });
await expect(cursor).toHaveCSS("opacity", "1");
await expect(cursor).toHaveAttribute("data-action", /click|type/);
const layer = (await preview.getByTestId("agent-cursor-layer").boundingBox())!;
await expect
.poll(async () => {
const tip = (await cursor.boundingBox())!;
return [
Math.round(tip.x - layer.x - layer.width / 2),
Math.round(tip.y - layer.y - layer.height / 2),
];
})
.toEqual([0, 0]);
await captureScreenshot(page, testInfo, "agent-cursor-on-screen");
// It fades once the bot stops pointing, so it never lingers over the screen.
await expect(cursor).toHaveCSS("opacity", "0", { timeout: 10_000 });
});

View File

@ -0,0 +1,432 @@
import type { Page, Request } from "@playwright/test";
/*
* Scripted Media Studio backend for the project and review specs. No RPC can
* seed an emulator library into a fresh space, so these specs answer the media
* procedures they use with synthetic, schema-shaped data and record every call.
*/
export const PROJECT_ID = "proj-e2e";
export const SHORTS = "youtube-shorts";
export const PAID = "example-paid-video";
export const TIKTOK = "tiktok";
const at = (minute: number) => new Date(Date.UTC(2026, 9, 1, 10, minute)).toISOString();
function clip(id: string, assetId: string, inMs: number, outMs: number, atMs: number) {
return {
id,
assetId,
inMs,
outMs,
atMs,
speed: 1,
reframe: "auto",
stabilize: false,
muted: false,
};
}
export function edl(profileId: string, width: number, height: number) {
return {
edlVersion: 1,
output: {
profileId,
width,
height,
fps: 30,
durationTargetMs: 12_000,
durationHardMinMs: 3_000,
durationHardMaxMs: 60_000,
},
tracks: {
video: [
clip("clip-1", "asset-1", 0, 4_000, 0),
clip("clip-2", "asset-2", 0, 4_000, 4_000),
clip("clip-3", "asset-3", 1_000, 4_000, 8_000),
],
audio: [],
},
transitions: [],
captions: { mode: "burned", maxCharsPerLine: 32, wordHighlight: false, position: "lower" },
overlays: [{ id: "text-1", kind: "text", atMs: 500, durationMs: 2_000, text: "Wait for it" }],
redactions: {
faces: [
{
id: "face-1",
target: { personId: "person-a" },
mode: "partial_eyes",
paddingPct: 20,
featherPct: 10,
minSizePct: 2,
},
],
marks: [{ markId: "mark-1", treatment: "cover" }],
regions: [],
},
metadata: { title: "Teaser", tags: [], aiAltered: false, aiAlterations: [] },
};
}
function plan(id: string, profileId: string, version: number, extra: Record<string, unknown> = {}) {
const [width, height] = profileId === PAID ? [1920, 1080] : [1080, 1920];
return {
id,
projectId: PROJECT_ID,
profileId,
version,
edl: edl(profileId, width, height),
createdBy: "user",
rationale: null,
createdAt: at(version),
...extra,
};
}
const report = (profileId: string, issues: unknown[], passed: boolean, autoFixRound = 0) => ({
profileId,
passed,
checkedAt: at(30),
sampledFrames: 48,
issues,
autoFixRound,
});
function render(id: string, extra: Record<string, unknown>) {
return {
id,
planId: "plan-shorts-1",
projectId: PROJECT_ID,
profileId: SHORTS,
quality: "draft",
status: "done",
progress: 1,
outputs: [
{ kind: "video", artifactId: null, fileName: "video.mp4", sizeBytes: 1_000_000 },
{ kind: "poster", artifactId: "art-poster", fileName: "poster.jpg", sizeBytes: 20_000 },
],
gateReport: null,
ffprobe: null,
error: null,
createdAt: at(20),
updatedAt: at(21),
...extra,
};
}
export const BLOCKED_ISSUES = [
{
check: "anonymization",
severity: "block",
code: "mask_coverage",
message: "A face of Alex is not fully covered.",
atMs: 5_000,
endMs: 5_600,
personId: "person-a",
},
{
check: "content",
severity: "block",
code: "classifier_threshold",
message: "Explicit nudity above the profile limit.",
atMs: 2_000,
category: "explicit_nudity",
score: 0.9,
},
{
check: "consent",
severity: "block",
code: "consent_missing",
message: "Sam has no recorded consent.",
atMs: 9_000,
personId: "person-b",
},
{
check: "anonymization",
severity: "warn",
code: "mark_residual",
message: "The tattoo is faintly visible.",
atMs: 10_000,
endMs: 10_400,
markId: "mark-1",
},
{
check: "text",
severity: "warn",
code: "hashtags",
message: "Fewer hashtags would read better.",
},
];
/** Mutable backend state the mock answers from; specs read `calls` to assert requests. */
export function mediaBackend() {
const plans: Record<string, unknown>[] = [
plan("plan-shorts-1", SHORTS, 1),
plan("plan-paid-1", PAID, 1),
plan("plan-paid-2", PAID, 2, {
createdBy: "agent",
rationale: "Auto-fix 1/3 after the gate blocked (mask_coverage): widened the face mask",
}),
];
const renders: Record<string, unknown>[] = [
render("render-shorts-draft", { gateReport: report(SHORTS, [], true) }),
render("render-paid-draft", {
planId: "plan-paid-1",
profileId: PAID,
status: "blocked",
outputs: [],
gateReport: report(PAID, BLOCKED_ISSUES, false, 1),
}),
];
const people = [
{
id: "person-a",
displayName: "Alex",
isSelf: false,
confirmed: true,
anonymizeByDefault: "partial_eyes",
consentRecordedAt: at(1),
consentArtifactId: null,
ageVerifiedAt: at(1),
ageVerificationMethod: "platform_verified",
faceTrackCount: 3,
coverThumbAssetId: null,
},
{
id: "person-b",
displayName: "Sam",
isSelf: false,
confirmed: true,
anonymizeByDefault: "blur",
consentRecordedAt: null,
consentArtifactId: null,
ageVerifiedAt: null,
ageVerificationMethod: "none",
faceTrackCount: 1,
coverThumbAssetId: null,
},
];
const marks = [
{
id: "mark-1",
kind: "tattoo",
label: null,
personId: "person-a",
assetId: "asset-2",
prompt: { tMs: 1_000, point: { x: 0.4, y: 0.6 } },
treatment: "inpaint",
qualityScore: 0.42,
suggestedFallback: "cover",
createdAt: at(2),
},
];
const calls: Array<{ procedure: string; input: Record<string, unknown> }> = [];
/** Submissions answered with these errors first, then accepted. */
const submitErrors: unknown[][] = [];
let renderCounter = 0;
let pollsUntilDone = 0;
const project = {
id: PROJECT_ID,
title: "Harbor teaser",
goal: "teaser",
status: "review",
brandKitId: null,
collectionIds: [],
profileIds: [SHORTS, PAID, TIKTOK],
createdAt: at(0),
updatedAt: at(0),
};
function advanceRenders() {
for (const item of renders) {
if (item.status !== "queued" && item.status !== "rendering") continue;
if (pollsUntilDone > 0) {
pollsUntilDone -= 1;
item.status = "rendering";
item.progress = 0.5;
continue;
}
item.status = "done";
item.progress = 1;
item.outputs = [
{ kind: "video", artifactId: null, fileName: "video.mp4", sizeBytes: 2_000_000 },
];
item.gateReport = report(String(item.profileId), [], true);
}
}
const profiles = [
[SHORTS, "YouTube Shorts", "free_social", false],
[PAID, "Example paid video", "paid_video", false],
[TIKTOK, "TikTok", "free_social", true],
].map(([id, name, kind, unverified]) => ({
id,
name,
kind,
origin: "builtin",
unverified,
stale: false,
aspectRatios: id === PAID ? ["16:9"] : ["9:16"],
hardMinMs: 1_000,
hardMaxMs: 60_000,
recommendedMs: [15_000],
}));
const handlers: Record<string, (input: Record<string, unknown>) => unknown> = {
capabilities: () => ({
configured: true,
operations: {},
accelerator: "cpu",
hardwareEncoders: [],
}),
"profiles/list": () => profiles,
"projects/get": () => {
advanceRenders();
return { project, plans, renders };
},
"projects/autoPlan": () => {
const created = plan("plan-tiktok-1", TIKTOK, 1, { createdBy: "agent" });
plans.push(created);
return [created];
},
"assets/get": (input) => ({
asset: {
id: input.assetId,
sourceId: "source-1",
externalId: `${String(input.assetId)}.mp4`,
kind: "video",
storage: "referenced",
capturedAt: at(0),
durationMs: 6_000,
width: 1920,
height: 1080,
fps: 30,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
ingestState: "analyzed",
hasProxy: true,
hasThumbnail: true,
qualityScore: 0.8,
createdAt: at(0),
},
segments: [],
faceTracks: [],
transcript: null,
}),
"plans/submit": (input) => {
const errors = submitErrors.shift();
if (errors) return { ok: false, errors };
const base = plans.find((item) => item.profileId === (input.edl as { output: { profileId: string } }).output.profileId);
const version = plans.filter((item) => item.profileId === base?.profileId).length + 1;
const created = {
...plan(`plan-saved-${version}`, String(base?.profileId), version),
edl: input.edl,
};
plans.push(created);
return { ok: true, plan: created };
},
"renders/start": (input) => {
const source = plans.find((item) => item.id === input.planId);
renderCounter += 1;
const created = render(`render-new-${renderCounter}`, {
planId: input.planId,
profileId: source?.profileId,
quality: input.quality,
status: "queued",
progress: 0,
outputs: [],
createdAt: at(40 + renderCounter),
});
renders.push(created);
return created;
},
"renders/get": (input) => {
advanceRenders();
return renders.find((item) => item.id === input.renderId);
},
"renders/thumbnails": () =>
["a", "b", "c"].map((letter) => ({
kind: "thumbnail",
artifactId: `art-thumb-${letter}`,
fileName: `thumb-${letter}.png`,
sizeBytes: 10_000,
})),
"marketing/copy": (input) => ({
profileId: SHORTS,
language: input.language,
title: "Harbor at dawn",
hook: "You have never seen the harbor like this",
caption: "Three minutes before sunrise.",
hashtags: ["#harbor", "#sunrise"],
issues: [],
}),
"people/list": () => people,
"people/update": (input) => {
const person = people.find((item) => item.id === input.personId);
if (person && typeof input.anonymizeByDefault === "string") {
person.anonymizeByDefault = input.anonymizeByDefault;
}
return person;
},
"marks/list": () => marks,
"marks/update": (input) => {
const mark = marks.find((item) => item.id === input.markId);
if (mark && typeof input.treatment === "string") mark.treatment = input.treatment;
return mark;
},
"marks/create": (input) => {
const created = {
id: `mark-${marks.length + 1}`,
kind: input.kind,
label: null,
personId: null,
assetId: input.assetId,
prompt: input.prompt,
treatment: input.treatment,
qualityScore: 0.9,
suggestedFallback: null,
createdAt: at(50),
};
marks.push(created as (typeof marks)[number]);
return created;
},
};
return {
calls,
submitErrors,
/** Let a started render report `rendering` for this many polls before it finishes. */
holdRenders(polls: number) {
pollsUntilDone = polls;
},
async install(page: Page) {
await page.route("**/rpc/media/**", async (route) => {
const request: Request = route.request();
const procedure = new URL(request.url()).pathname.replace(/^.*\/rpc\/media\//, "");
const handler = handlers[procedure];
const body = (request.postDataJSON() ?? {}) as { json?: Record<string, unknown> };
const input = body.json ?? {};
calls.push({ procedure, input });
if (!handler) {
await route.fulfill({
status: 404,
contentType: "application/json",
body: JSON.stringify({ json: { message: `unmocked ${procedure}` } }),
});
return;
}
await route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({ json: handler(input) }),
});
});
// Byte routes: no synthetic media files are needed for these flows.
await page.route("**/api/media/**", (route) => route.fulfill({ status: 404, body: "" }));
},
};
}

View File

@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Media editor test</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/e2e/fixtures/media-editor.tsx"></script>
</body>
</html>

View File

@ -0,0 +1,58 @@
import { setupI18n } from "@lingui/core";
import { I18nProvider } from "@lingui/react";
import { useState } from "react";
import { createRoot } from "react-dom/client";
import { BrandKitView } from "../../src/pages/media/BrandKitView";
import { EditorView } from "../../src/pages/media/EditorView";
import type { MediaView } from "../../src/pages/media/nav";
import { ProjectView } from "../../src/pages/media/ProjectView";
import { ReviewView } from "../../src/pages/media/ReviewView";
import "../../src/styles.css";
/*
* Mounts the Media Studio editor-side views on their own, starting at the view
* named in the query string, so specs reach them without the library flow.
* Views the editor side does not own render as a marker the spec can assert.
*/
const params = new URLSearchParams(location.search);
const i18n = setupI18n({ locale: "en", messages: { en: {} } });
function initialView(): MediaView {
const projectId = params.get("projectId") ?? "";
switch (params.get("view")) {
case "editor":
return { name: "editor", projectId, planId: params.get("planId") ?? "" };
case "review":
return { name: "review", projectId, renderId: params.get("renderId") ?? "" };
case "brandKit":
return { name: "brandKit", brandKitId: params.get("brandKitId") ?? undefined };
default:
return { name: "project", projectId };
}
}
function Harness() {
const [view, setView] = useState<MediaView>(initialView);
const props = { navigate: setView, onClose: () => undefined };
switch (view.name) {
case "project":
return <ProjectView {...props} projectId={view.projectId} />;
case "editor":
return <EditorView {...props} projectId={view.projectId} planId={view.planId} />;
case "review":
return <ReviewView {...props} projectId={view.projectId} renderId={view.renderId} />;
case "brandKit":
return <BrandKitView {...props} brandKitId={view.brandKitId} />;
default:
return <div data-testid="media-other-view" data-view={view.name} />;
}
}
createRoot(document.getElementById("root")!).render(
<I18nProvider i18n={i18n}>
<main className="min-h-screen bg-background text-foreground">
<Harness />
</main>
</I18nProvider>,
);

View File

@ -0,0 +1,159 @@
import { expect, test } from "@playwright/test";
import { captureScreenshot, completeOnboarding, signup } from "./helpers";
test("the in-app browser previews a page and follows it back to the browser", async ({
page,
}, testInfo) => {
await signup(page, `in-app-browser-${Date.now()}@rakazo.test`, "password12", "Previewer");
await completeOnboarding(page);
// Context-wide so the "Open in browser" tab is served too.
await page.context().route("https://preview.example/**", (route) =>
route.fulfill({
contentType: "text/html",
body: `<!doctype html><title>Preview</title><body style="font:16px system-ui;padding:24px">
<h1>Live preview</h1><p>Rendered inside the app.</p>`,
}),
);
await page.getByRole("button", { name: "Browser" }).click();
const browser = page.getByTestId("in-app-browser");
const address = browser.getByRole("textbox", { name: "Address" });
await expect(address).toBeFocused();
await address.fill("not an address");
await address.press("Enter");
await expect(address).toHaveAttribute("aria-invalid", "true");
await address.fill("preview.example/app");
await address.press("Enter");
await expect(address).toHaveValue("https://preview.example/app");
const frame = browser.getByTestId("in-app-browser-frame");
await expect(frame).toHaveAttribute("src", "https://preview.example/app");
await expect(
page.frameLocator('[data-testid="in-app-browser-frame"]').getByRole("heading"),
).toHaveText("Live preview");
await captureScreenshot(page, testInfo, "in-app-browser");
const popup = page.waitForEvent("popup");
await browser.getByRole("button", { name: "Open in browser" }).click();
expect((await popup).url()).toBe("https://preview.example/app");
await browser.getByRole("button", { name: "Close panel" }).click();
await expect(browser).toHaveCount(0);
// The last address comes back when the browser reopens.
await page.getByRole("button", { name: "Browser" }).click();
await expect(page.getByTestId("in-app-browser-frame")).toHaveAttribute(
"src",
"https://preview.example/app",
);
});
test("on desktop the page is a native view glued to the panel", async ({ page }) => {
await page.addInitScript(() => {
const calls: Array<[string, unknown?]> = [];
let listener: ((state: unknown) => void) | undefined;
let state = {
url: "",
title: "",
loading: false,
canGoBack: false,
canGoForward: false,
error: null as string | null,
};
const record = (name: string) => async (value?: unknown) => {
calls.push([name, value]);
};
Object.assign(window, { __browserCalls: calls });
Object.defineProperty(window, "rakazoDesktop", {
value: {
platform: "linux",
window: {
close: async () => {},
minimize: async () => {},
toggleMaximize: async () => {},
state: async () => ({ minimized: false, maximized: false, fullScreen: false }),
},
oauth: { onCallback: () => () => {} },
update: {
state: async () => ({ phase: "unsupported", currentVersion: "0.0.0" }),
check: async () => ({ phase: "unsupported", currentVersion: "0.0.0" }),
download: async () => ({ phase: "unsupported", currentVersion: "0.0.0" }),
install: async () => ({ phase: "unsupported", currentVersion: "0.0.0" }),
},
browserPreview: {
state: async () => state,
onState: (next: (value: unknown) => void) => {
listener = next;
return () => {
listener = undefined;
};
},
navigate: async (url: string) => {
calls.push(["navigate", url]);
state = { ...state, url, title: "Dev server", loading: false, canGoBack: true };
listener?.(state);
},
show: record("show"),
hide: record("hide"),
back: record("back"),
forward: record("forward"),
reload: record("reload"),
stop: record("stop"),
openDevTools: record("openDevTools"),
},
},
});
});
await signup(page, `native-browser-${Date.now()}@rakazo.test`, "password12", "Desktop");
await completeOnboarding(page);
const calls = () =>
page.evaluate(
() => (window as unknown as { __browserCalls: Array<[string, unknown?]> }).__browserCalls,
);
await page.getByRole("button", { name: "Browser" }).click();
const browser = page.getByTestId("in-app-browser");
await browser.getByRole("textbox", { name: "Address" }).fill("localhost:5173");
await browser.getByRole("textbox", { name: "Address" }).press("Enter");
await expect.poll(calls).toContainEqual(["navigate", "http://localhost:5173/"]);
await expect(browser.getByRole("button", { name: "Back" })).toBeEnabled();
await expect(browser.getByRole("button", { name: "Forward" })).toBeDisabled();
await expect(browser.getByTestId("in-app-browser-frame")).toHaveCount(0);
// The reserved page area and the native view's bounds stay identical.
const viewport = browser.getByTestId("in-app-browser-viewport");
await expect
.poll(async () => {
const box = (await viewport.boundingBox())!;
const shown = (await calls()).filter(([name]) => name === "show").at(-1)?.[1];
return (
JSON.stringify(shown) ===
JSON.stringify({
x: Math.round(box.x),
y: Math.round(box.y),
width: Math.round(box.width),
height: Math.round(box.height),
})
);
})
.toBe(true);
await browser.getByRole("button", { name: "Back" }).click();
await browser.getByRole("button", { name: "Reload" }).click();
await browser.getByRole("button", { name: "Developer tools" }).click();
await expect
.poll(async () => (await calls()).map(([name]) => name))
.toEqual(expect.arrayContaining(["back", "reload", "openDevTools"]));
// App menus float above the page, so the native view steps aside while one is open.
const before = (await calls()).length;
await page.getByTestId("create-menu-trigger").click();
await expect
.poll(async () => (await calls()).slice(before).map(([name]) => name))
.toContain("hide");
await page.keyboard.press("Escape");
await expect.poll(async () => (await calls()).at(-1)?.[0]).toBe("show");
await browser.getByRole("button", { name: "Close panel" }).click();
await expect.poll(async () => (await calls()).at(-1)?.[0]).toBe("hide");
});

View File

@ -0,0 +1,60 @@
import { expect, test } from "@playwright/test";
import { captureScreenshot, completeOnboarding, rpc, signup } from "./helpers";
test.setTimeout(180_000);
test("brand kit editor saves a kit and the intro generator offers three variants", async ({
page,
}, testInfo) => {
const stamp = Date.now();
await signup(page, `media-brand-${stamp}@rakazo.test`, "password12", "Brand Kit");
await completeOnboarding(page);
// The scripted runtime runs Media Studio on the inference emulator.
const capabilities = await rpc<{ configured: boolean }>(page, "media/capabilities", {});
expect(capabilities.configured).toBe(true);
await page.goto("/e2e/fixtures/media-editor.html?view=brandKit");
const editor = page.getByTestId("media-brand-kit");
await expect(editor).toBeVisible();
// A new kit has nothing to generate an intro from until it is saved.
await expect(editor.getByTestId("media-intro-generator")).toHaveCount(0);
await editor.getByLabel("Name").fill("Studio Nord");
await editor.getByLabel("Tone").fill("calm, precise");
await editor.getByLabel("Primary").fill("#1f4e79");
await editor.getByRole("switch", { name: "Watermark" }).click();
await editor.getByLabel("Position").selectOption("top_right");
await editor.getByRole("button", { name: "Caption style" }).click();
await editor.getByLabel("Look").selectOption("karaoke");
await editor.getByRole("button", { name: "Save" }).click();
const generator = editor.getByTestId("media-intro-generator");
await expect(generator).toBeVisible();
await expect(editor.getByRole("button", { name: "Save" })).toBeDisabled();
await expect(editor.getByRole("heading", { name: "Studio Nord" })).toBeVisible();
await captureScreenshot(page, testInfo, "media-brand-kit");
await generator.getByRole("button", { name: "Suggest" }).click();
const variants = generator.getByTestId("media-intro-variant");
await expect(variants).toHaveCount(3, { timeout: 120_000 });
await captureScreenshot(page, testInfo, "media-brand-kit-intros");
await variants.first().getByRole("button", { name: "Use" }).click();
await expect(variants.first().getByText("In use")).toBeVisible();
await expect(variants.nth(1).getByRole("button", { name: "Use" })).toBeVisible();
// The intro is stored on the kit.
const kits = await rpc<Array<{ kit: { name: string; intro?: { templateId: string } } }>>(
page,
"media/brandKits/list",
{},
);
const saved = kits.find((record) => record.kit.name === "Studio Nord");
expect(saved?.kit.intro?.templateId).toBeTruthy();
// Back lists the kits.
await editor.getByRole("button", { name: "Back" }).click();
const list = page.getByTestId("media-brand-kits");
await expect(list.getByText("Studio Nord")).toBeVisible();
await captureScreenshot(page, testInfo, "media-brand-kit-list");
});

View File

@ -0,0 +1,144 @@
import { expect, test } from "@playwright/test";
import type { edl } from "./fixtures/media-editor-data";
import { mediaBackend, PAID, PROJECT_ID, SHORTS, TIKTOK } from "./fixtures/media-editor-data";
import { captureScreenshot } from "./helpers";
const projectUrl = `/e2e/fixtures/media-editor.html?view=project&projectId=${PROJECT_ID}`;
test("project variants plan, approve into a final render, and publish", async ({
page,
}, testInfo) => {
const backend = mediaBackend();
await backend.install(page);
await page.goto(projectUrl);
const project = page.getByTestId("media-project");
await expect(project.getByRole("heading", { name: "Harbor teaser" })).toBeVisible();
const shorts = page.locator(`[data-testid="media-variant"][data-profile="${SHORTS}"]`);
const paid = page.locator(`[data-testid="media-variant"][data-profile="${PAID}"]`);
const tiktok = page.locator(`[data-testid="media-variant"][data-profile="${TIKTOK}"]`);
await expect(shorts.getByTestId("media-gate-status")).toHaveAttribute("data-status", "pass");
await expect(paid.getByTestId("media-gate-status")).toHaveAttribute("data-status", "blocked");
await expect(paid.getByText("Auto-fix 1/3")).toBeVisible();
await expect(tiktok.getByText("No plan yet")).toBeVisible();
await expect(tiktok.getByText("Unverified")).toBeVisible();
// A blocked draft cannot be approved.
await expect(paid.getByRole("button", { name: "Approve" })).toHaveCount(0);
await captureScreenshot(page, testInfo, "media-project-variants");
// Auto-plan fills the profile without a plan.
await project.getByRole("button", { name: "Plan" }).click();
await expect(tiktok.getByRole("button", { name: "Edit" })).toBeVisible();
await expect(project.getByRole("button", { name: "Plan" })).toHaveCount(0);
// Approving a passed draft starts the final render; the card polls until it is done.
backend.holdRenders(2);
await shorts.getByRole("button", { name: "Approve" }).click();
await expect(shorts.getByText(/Rendering \d+%/)).toBeVisible();
const start = backend.calls.find((call) => call.procedure === "renders/start");
expect(start?.input).toEqual({ planId: "plan-shorts-1", quality: "final" });
await expect(shorts.getByTestId("media-publish")).toBeVisible({ timeout: 15_000 });
await expect(shorts.getByRole("button", { name: "Download" })).toBeVisible();
// Polling stops once nothing renders.
const polls = backend.calls.filter((call) => call.procedure === "projects/get").length;
await page.waitForTimeout(2_500);
expect(backend.calls.filter((call) => call.procedure === "projects/get").length).toBe(polls);
// Thumbnails, copy, and the bundle step appear in order.
const publish = shorts.getByTestId("media-publish");
await publish.getByRole("button", { name: "Thumbnails" }).click();
await expect(publish.getByRole("button", { name: /thumb-/ })).toHaveCount(3);
await expect(publish.getByRole("button", { name: "thumb-a" })).toHaveAttribute(
"aria-pressed",
"true",
);
await publish.getByRole("button", { name: "thumb-b" }).click();
await publish.getByRole("button", { name: "Write copy" }).click();
await expect(publish.getByRole("textbox", { name: "Title" })).toHaveValue("Harbor at dawn");
await expect(publish.getByRole("textbox", { name: "Hashtags" })).toHaveValue(
"#harbor #sunrise",
);
await expect(publish.getByRole("button", { name: "Bundle" })).toBeVisible();
await captureScreenshot(page, testInfo, "media-project-publish");
});
test("the editor trims and moves clips, edits captions and overlays, and shows EDL errors", async ({
page,
}, testInfo) => {
const backend = mediaBackend();
await backend.install(page);
backend.submitErrors.push([
{
code: "overlap",
path: "tracks.video[1].atMs",
message: "clip-2 overlaps clip-3 on the video track",
},
{ code: "duration_above_max", path: "output", message: "duration is above the limit" },
]);
await page.goto(projectUrl);
const shorts = page.locator(`[data-testid="media-variant"][data-profile="${SHORTS}"]`);
await shorts.getByRole("button", { name: "Edit" }).click();
const editor = page.getByTestId("media-editor");
await expect(editor).toBeVisible();
const clips = editor.getByTestId("media-clip");
await expect(clips).toHaveCount(3);
await expect(editor.getByRole("button", { name: "Save" })).toBeDisabled();
// Trim the first clip's end to the right with the pointer.
const trimEnd = clips.first().getByTestId("media-trim-end");
const handle = await trimEnd.boundingBox();
if (!handle) throw new Error("trim handle not laid out");
await page.mouse.move(handle.x + handle.width / 2, handle.y + handle.height / 2);
await page.mouse.down();
await page.mouse.move(handle.x + 40, handle.y + handle.height / 2, { steps: 5 });
await page.mouse.move(handle.x + 60, handle.y + handle.height / 2, { steps: 5 });
await page.mouse.up();
// Move the third clip later.
const third = clips.nth(2).getByRole("button", { name: "Clip clip-3" });
const box = await third.boundingBox();
if (!box) throw new Error("clip not laid out");
await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2);
await page.mouse.down();
await page.mouse.move(box.x + box.width / 2 + 30, box.y + box.height / 2, { steps: 5 });
await page.mouse.move(box.x + box.width / 2 + 50, box.y + box.height / 2, { steps: 5 });
await page.mouse.up();
// Captions: switch to the subtitle file and highlight words.
await editor.getByRole("button", { name: "Captions", exact: true }).click();
const inspector = editor.getByTestId("media-inspector");
await inspector.getByLabel("Captions").selectOption("both");
await inspector.getByRole("switch", { name: "Highlight words" }).click();
// A new text overlay at the playhead.
await editor.getByRole("button", { name: "Text", exact: true }).click();
await inspector.getByLabel("Text").fill("Link in bio");
await editor.getByRole("button", { name: "Save" }).click();
const errors = editor.getByTestId("media-edl-errors");
await expect(errors.getByText("clip-2 overlaps clip-3 on the video track")).toBeVisible();
await expect(errors.getByText("duration is above the limit")).toBeVisible();
await expect(clips.nth(1)).toHaveAttribute("data-invalid", "true");
await captureScreenshot(page, testInfo, "media-editor-errors");
const submitted = backend.calls.find((call) => call.procedure === "plans/submit")?.input as {
projectId: string;
edl: ReturnType<typeof edl>;
};
expect(submitted.projectId).toBe(PROJECT_ID);
const [first, , moved] = submitted.edl.tracks.video;
expect(first?.outMs).toBeGreaterThan(4_000);
expect(first?.outMs).toBeLessThanOrEqual(6_000);
expect(moved?.atMs).toBeGreaterThan(8_000);
expect(submitted.edl.captions).toMatchObject({ mode: "both", wordHighlight: true });
expect(submitted.edl.overlays.at(-1)).toMatchObject({ kind: "text", text: "Link in bio" });
// Selecting an error selects the clip; the next save is accepted as a new version.
await errors.getByText("clip-2 overlaps clip-3 on the video track").click();
await expect(inspector.getByRole("spinbutton", { name: "In", exact: true })).toBeVisible();
await editor.getByRole("button", { name: "Save" }).click();
await expect(editor.getByRole("heading", { name: `${SHORTS} · v2` })).toBeVisible();
await expect(errors).toHaveCount(0);
await captureScreenshot(page, testInfo, "media-editor");
});

View File

@ -0,0 +1,117 @@
import { expect, test } from "@playwright/test";
import { mediaBackend, PROJECT_ID } from "./fixtures/media-editor-data";
import { captureScreenshot } from "./helpers";
const reviewUrl = (renderId: string) =>
`/e2e/fixtures/media-editor.html?view=review&projectId=${PROJECT_ID}&renderId=${renderId}`;
test("review lists gate blockers that jump the player and shows coverage and auto-fix", async ({
page,
}, testInfo) => {
const backend = mediaBackend();
await backend.install(page);
await page.goto(reviewUrl("render-paid-draft"));
const review = page.getByTestId("media-review");
await expect(review.getByTestId("media-gate-status")).toHaveAttribute("data-status", "blocked");
await expect(review.getByTestId("media-autofix")).toContainText("Auto-fix 1/3");
await expect(review.getByTestId("media-autofix")).toContainText("widened the face mask");
// Issues in timeline order, untimed last.
const issues = review.getByTestId("media-gate-issue");
await expect(issues).toHaveCount(5);
await expect(issues.nth(0)).toContainText("0:02.0");
await expect(issues.nth(0)).toContainText("Explicit nudity above the profile limit.");
await expect(issues.nth(1)).toContainText("0:05.0");
await expect(issues.nth(4)).toContainText("Fewer hashtags would read better.");
await expect(issues.nth(4).getByRole("button")).toBeDisabled();
// A blocked render has no output; the plan plays from the sources.
const player = review.getByTestId("media-source-player");
await expect(review.getByRole("button", { name: "After" })).toHaveCount(0);
await issues.nth(1).getByRole("button").click();
await expect(player).toHaveAttribute("data-position-ms", "5000");
await issues.nth(2).getByRole("button").click();
await expect(player).toHaveAttribute("data-position-ms", "9000");
// Coverage per person and mark; a segment seeks too.
const coverage = review.getByTestId("media-coverage");
await expect(coverage.getByText("Alex")).toBeVisible();
await expect(coverage.getByText("Sam")).toBeVisible();
await expect(coverage.getByText("Tattoo")).toBeVisible();
await coverage.getByRole("button", { name: /0:10\.0/ }).click();
await expect(player).toHaveAttribute("data-position-ms", "10000");
await captureScreenshot(page, testInfo, "media-review");
});
test("review edits people, marks, and masks, then re-renders the draft", async ({
page,
}, testInfo) => {
const backend = mediaBackend();
await backend.install(page);
await page.goto(reviewUrl("render-paid-draft"));
const review = page.getByTestId("media-review");
await expect(review.getByTestId("media-gate-issue")).toHaveCount(5);
// Per person: anonymization mode; consent only where the gate asks for it.
const people = review.getByTestId("media-review-person");
await expect(people).toHaveCount(2);
await review.getByLabel("Anonymization for Alex").selectOption("pixelate");
await expect
.poll(() => backend.calls.find((call) => call.procedure === "people/update")?.input)
.toEqual({ personId: "person-a", anonymizeByDefault: "pixelate" });
await expect(people.nth(0).getByRole("button", { name: "Record consent" })).toHaveCount(0);
await expect(people.nth(1).getByRole("button", { name: "Record consent" })).toBeVisible();
// Per mark: a low quality score offers the suggested fallback.
const mark = review.getByTestId("media-review-mark");
await expect(mark).toContainText("42%");
await mark.getByRole("button", { name: "Use Cover" }).click();
await expect(mark.getByLabel("Treatment")).toHaveValue("cover");
await expect(mark.getByRole("button", { name: "Use Cover" })).toHaveCount(0);
// Mark a spot on the frame.
await review.getByRole("button", { name: "Mark", exact: true }).click();
await review.getByRole("button", { name: "Mark this spot" }).click({ position: { x: 20, y: 20 } });
await expect
.poll(() => backend.calls.find((call) => call.procedure === "marks/create")?.input)
.toMatchObject({ kind: "custom", assetId: "asset-1", treatment: "blur" });
// Draw a mask and drag it.
await review.getByRole("button", { name: "Mask", exact: true }).click();
const mask = review.getByTestId("media-mask");
await expect(mask).toBeVisible();
const box = await mask.boundingBox();
if (!box) throw new Error("mask not laid out");
await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2);
await page.mouse.down();
await page.mouse.move(box.x + box.width / 2 + 40, box.y + box.height / 2 + 10, { steps: 4 });
await page.mouse.up();
await captureScreenshot(page, testInfo, "media-review-mask");
await review.getByRole("button", { name: "Save" }).click();
await expect(review.getByRole("button", { name: "Draft" })).toBeVisible();
const submitted = backend.calls.find((call) => call.procedure === "plans/submit")?.input as {
edl: {
redactions: {
regions: Array<{ clipId: string; keyframes: Array<{ box: { x: number } }> }>;
};
};
};
const region = submitted.edl.redactions.regions[0];
expect(region?.clipId).toBe("clip-1");
expect(region?.keyframes[0]?.box.x).toBeGreaterThan(0.4);
// Re-rendering opens the new render, which polls while it runs.
backend.holdRenders(1);
await review.getByRole("button", { name: "Draft" }).click();
await expect(review.getByText(/Rendering \d+%/)).toBeVisible();
await expect(review.getByTestId("media-gate-status")).toHaveAttribute("data-status", "pass", {
timeout: 15_000,
});
// Consent is recorded on the people screen, never here.
await page.goto(reviewUrl("render-paid-draft"));
await page.getByRole("button", { name: "Record consent" }).click();
await expect(page.getByTestId("media-other-view")).toHaveAttribute("data-view", "people");
});

View File

@ -50,6 +50,16 @@ test("live page helper preserves identity, masks passwords, and reports partial
expect(snapshot.title).toBe("Fixture");
expect(snapshot.tree).toContain("Readable article content");
expect(JSON.stringify(snapshot)).not.toContain("fake-test-password");
// Element boxes are screen pixels for the visible bot cursor.
expect(snapshot.screen.width).toBeGreaterThan(0);
expect(snapshot.screen.height).toBeGreaterThan(0);
const saveBox = snapshot.elements.find((element: { name: string }) => element.name === "Save")
.box as number[];
const saveRect = (await page.locator("#save").boundingBox())!;
expect(saveBox[2]).toBe(Math.round(saveRect.width));
expect(saveBox[3]).toBe(Math.round(saveRect.height));
expect(saveBox[0]).toBeGreaterThanOrEqual(Math.round(saveRect.x));
expect(saveBox[1]).toBeGreaterThanOrEqual(Math.round(saveRect.y));
const ref = (snapshot: { elements: { name: string; ref: string }[] }, name: string) =>
snapshot.elements.find((element) => element.name === name)!.ref;
const result = await command("act", {

View File

@ -3,6 +3,7 @@ import {
captureScreenshot,
completeOnboarding,
openNewSpace,
routePatchRpc,
rpc,
signup,
unterhaltungenOnly,
@ -45,6 +46,30 @@ test("Projekte + opens new project dialog instead of group form", async ({ page
await expect(page.getByTestId("project-start-main")).toHaveText(/Starten/);
});
for (const owner of [true, false]) {
const role = owner ? "owner" : "member";
const title = owner
? "new project dialog offers a host folder to the deployment owner"
: "new project dialog offers no host folder to a member";
test(title, async ({ page }, testInfo) => {
await signup(page, `proj-folder-${role}-${Date.now()}@rakazo.test`, "password12", "Folder");
await completeOnboarding(page);
// Whoever signs up first owns the deployment, which depends on spec order.
await routePatchRpc(page, "bootstrap", (json) => ({
...json,
me: { ...(json.me as Record<string, unknown>), isDeploymentOwner: owner },
}));
await page.goto("/app");
await page.waitForURL(/\/app\/[^/]+$/);
await page.locator(".sidebar-projekte").getByTestId("create-project-trigger").click();
const dialog = page.getByRole("dialog", { name: "New project" });
await expect(dialog.getByLabel("Name")).toBeVisible();
await expect(dialog.getByLabel("Project folder")).toHaveCount(owner ? 1 : 0);
await captureScreenshot(page, testInfo, `new-project-${role}`);
});
}
test("spaces stay invisible by default and chat creation requires approval", async ({
page,
}, testInfo) => {

View File

@ -24,6 +24,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/ui-tokens": "workspace:*",
"@rakazo/ui-web": "workspace:*",
"@tanstack/react-virtual": "3.14.13",
"better-auth": "1.7.4",
"lucide-react": "1.44.0",
"react": "19.2.3",

View File

@ -0,0 +1,317 @@
import { useLingui } from "@lingui/react/macro";
import type {
RakazoDesktopBrowserPreview,
RakazoDesktopBrowserPreviewState,
} from "@rakazo/contracts";
import { Button, cn, Input } from "@rakazo/ui-web";
import { ArrowLeft, ArrowRight, Code, ExternalLink, RotateCw, X } from "lucide-react";
import { type FormEvent, useCallback, useEffect, useRef, useState } from "react";
import { browserFrameSandbox, normalizeBrowserUrl } from "../../lib/browser-url";
import { desktopBridge } from "../../lib/desktop";
const LAST_URL_KEY = "rk:in-app-browser:url";
function readLastUrl(): string {
try {
return window.localStorage.getItem(LAST_URL_KEY) ?? "";
} catch {
return "";
}
}
function rememberUrl(url: string) {
try {
window.localStorage.setItem(LAST_URL_KEY, url);
} catch {
/* A per-viewer convenience only. */
}
}
const EMPTY_STATE: RakazoDesktopBrowserPreviewState = {
url: "",
title: "",
loading: false,
canGoBack: false,
canGoForward: false,
error: null,
};
/** DOM layers the native desktop view would otherwise paint over. */
const FLOATING_UI = '[role="dialog"],[role="alertdialog"],[role="menu"],[role="listbox"]';
/**
* Address bar plus page. On desktop the page is a native Chromium view laid over the reserved
* area, so dev servers keep hot reload, DevTools and full speed; in a browser it is an iframe.
*/
export function InAppBrowser({ onClose }: { onClose: () => void }) {
const { t } = useLingui();
const native = desktopBridge()?.browserPreview;
const [state, setState] = useState<RakazoDesktopBrowserPreviewState>(() => ({
...EMPTY_STATE,
url: native ? "" : readLastUrl(),
}));
const [draft, setDraft] = useState(state.url);
const [invalid, setInvalid] = useState(false);
const [frameKey, setFrameKey] = useState(0);
const editing = useRef(false);
const inputRef = useRef<HTMLInputElement>(null);
useEffect(() => {
if (!native) return;
let live = true;
const apply = (next: RakazoDesktopBrowserPreviewState) => {
if (!live) return;
setState(next);
if (!editing.current) setDraft(next.url);
};
const unsubscribe = native.onState(apply);
void native
.state()
.then((current) => {
apply(current);
const last = readLastUrl();
if (!current.url && last) void native.navigate(last).catch(() => undefined);
})
.catch(() => undefined);
return () => {
live = false;
unsubscribe();
};
}, [native]);
useEffect(() => {
if (!state.url) inputRef.current?.focus();
}, [state.url]);
const go = useCallback(
(input: string) => {
const url = normalizeBrowserUrl(input);
setInvalid(!url);
if (!url) return;
editing.current = false;
setDraft(url);
rememberUrl(url);
if (native) {
void native.navigate(url).catch(() => setInvalid(true));
} else {
setState((current) => ({ ...current, url, loading: true, error: null }));
setFrameKey((key) => key + 1);
}
},
[native],
);
const reload = useCallback(() => {
if (native) {
void (state.loading ? native.stop() : native.reload()).catch(() => undefined);
return;
}
if (!state.url) return;
setState((current) => ({ ...current, loading: true }));
setFrameKey((key) => key + 1);
}, [native, state.loading, state.url]);
const onSubmit = (event: FormEvent) => {
event.preventDefault();
go(draft);
inputRef.current?.blur();
};
return (
<div className="flex h-full min-h-0 flex-col" data-testid="in-app-browser">
<form onSubmit={onSubmit} className="relative flex items-center gap-0.5 border-b px-2 py-1.5">
{native ? (
<>
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={t`Back`}
disabled={!state.canGoBack}
onClick={() => void native.back().catch(() => undefined)}
>
<ArrowLeft strokeWidth={1.7} className="size-icon-md" />
</Button>
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={t`Forward`}
disabled={!state.canGoForward}
onClick={() => void native.forward().catch(() => undefined)}
>
<ArrowRight strokeWidth={1.7} className="size-icon-md" />
</Button>
</>
) : null}
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={state.loading ? t`Stop` : t`Reload`}
disabled={!state.url}
onClick={reload}
>
{state.loading && native ? (
<X strokeWidth={1.8} className="size-icon-md" />
) : (
<RotateCw strokeWidth={1.7} className="size-icon-md" />
)}
</Button>
<Input
ref={inputRef}
value={draft}
aria-label={t`Address`}
aria-invalid={invalid || undefined}
inputMode="url"
autoComplete="off"
autoCapitalize="off"
spellCheck={false}
placeholder="localhost:5173"
onFocus={(event) => {
editing.current = true;
event.currentTarget.select();
}}
onBlur={() => {
editing.current = false;
}}
onChange={(event) => {
setDraft(event.target.value);
setInvalid(false);
}}
onKeyDown={(event) => {
if (event.key !== "Escape") return;
setDraft(state.url);
setInvalid(false);
event.currentTarget.blur();
}}
className="mx-1 h-control-sm flex-1 rounded-full bg-muted/60 px-3 text-ui-body-sm"
/>
{native ? (
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={t`Developer tools`}
disabled={!state.url}
onClick={() => void native.openDevTools().catch(() => undefined)}
>
<Code strokeWidth={1.7} className="size-icon-md" />
</Button>
) : null}
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={t`Open in browser`}
disabled={!state.url}
onClick={() => window.open(state.url, "_blank", "noopener,noreferrer")}
>
<ExternalLink strokeWidth={1.7} className="size-icon-md" />
</Button>
<Button
type="button"
variant="ghost"
size="icon-sm"
aria-label={t`Close panel`}
onClick={onClose}
>
<X strokeWidth={1.8} className="size-icon-md" />
</Button>
<div
aria-hidden
className={cn(
"absolute inset-x-0 -bottom-px h-0.5 overflow-hidden transition-opacity duration-300",
state.loading ? "opacity-100" : "opacity-0",
)}
>
<div className="rk-browser-progress h-full w-1/3 rounded-full bg-primary/70" />
</div>
</form>
<div className="relative min-h-0 flex-1 bg-background">
{native ? (
<NativeViewport browser={native} visible={Boolean(state.url) && !state.error} />
) : state.url ? (
<iframe
key={frameKey}
title={state.title || t`Page`}
src={state.url}
sandbox={browserFrameSandbox(state.url, window.location.origin)}
allow="clipboard-read; clipboard-write; fullscreen"
onLoad={() => setState((current) => ({ ...current, loading: false }))}
className="h-full w-full border-0 bg-background"
data-testid="in-app-browser-frame"
/>
) : null}
{state.error ? (
<div
role="alert"
className="absolute inset-0 grid place-items-center px-6 text-center text-ui-body-sm text-muted-foreground"
>
{state.error}
</div>
) : null}
</div>
</div>
);
}
/**
* Reserves the page area and keeps the native view glued to it while the panel resizes,
* hiding it whenever app UI (dialogs, menus) floats where the view would paint over it.
*/
function NativeViewport({
browser,
visible,
}: {
browser: RakazoDesktopBrowserPreview;
visible: boolean;
}) {
const ref = useRef<HTMLDivElement>(null);
useEffect(() => {
const element = ref.current;
if (!element) return;
let last = "";
let frame = 0;
const sync = () => {
frame = 0;
const rect = element.getBoundingClientRect();
const covered = Array.from(document.querySelectorAll(FLOATING_UI)).some(
(node) => !element.contains(node),
);
const show = visible && rect.width >= 1 && rect.height >= 1 && !covered;
const bounds = {
x: Math.round(rect.left),
y: Math.round(rect.top),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
const key = show ? JSON.stringify(bounds) : "hidden";
if (key === last) return;
last = key;
void (show ? browser.show(bounds) : browser.hide()).catch(() => undefined);
};
const schedule = () => {
if (!frame) frame = requestAnimationFrame(sync);
};
const resize = new ResizeObserver(schedule);
resize.observe(element);
const floating = new MutationObserver(schedule);
floating.observe(document.body, { childList: true, subtree: true });
window.addEventListener("resize", schedule);
// Sibling layout (sidebars, transitions) can move the area without resizing it.
const settle = setInterval(schedule, 250);
schedule();
return () => {
cancelAnimationFrame(frame);
resize.disconnect();
floating.disconnect();
window.removeEventListener("resize", schedule);
clearInterval(settle);
void browser.hide().catch(() => undefined);
};
}, [browser, visible]);
return <div ref={ref} className="absolute inset-0" data-testid="in-app-browser-viewport" />;
}

View File

@ -0,0 +1,165 @@
import { screenPointInViewer } from "@rakazo/core";
import { Keyboard } from "lucide-react";
import { useEffect, useRef, useState } from "react";
import {
agentPointerFeed,
POINTER_IDLE_MS,
type PointerPoint,
type PointerScreen,
type PointerSegment,
planPointerTimeline,
pointerFrameAt,
} from "../../lib/agent-pointer";
function prefersReducedMotion() {
return (
typeof window.matchMedia === "function" &&
window.matchMedia("(prefers-reduced-motion: reduce)").matches
);
}
/**
* Draws a bot's announced pointer over its live screen: it glides to each target, ripples on
* clicks and shows when it types. Frames are written straight to the DOM so the chat never
* re-renders at animation rate.
*/
export function AgentCursor({
botId,
color,
name,
}: {
botId: string;
color: string;
name: string;
}) {
const layerRef = useRef<HTMLDivElement>(null);
const cursorRef = useRef<HTMLDivElement>(null);
const rippleRef = useRef<HTMLSpanElement>(null);
const [typing, setTyping] = useState(false);
useEffect(() => {
const layer = layerRef.current;
const cursor = cursorRef.current;
const ripple = rippleRef.current;
if (!layer || !cursor || !ripple) return;
const reducedMotion = prefersReducedMotion();
let screen: PointerScreen | null = null;
let segments: PointerSegment[] = [];
let position: PointerPoint | null = null;
let frame = 0;
let idleTimer: ReturnType<typeof setTimeout> | undefined;
let viewer = { width: layer.clientWidth, height: layer.clientHeight };
let shownTyping = false;
const place = (point: PointerPoint, action: string | null, progress: number) => {
if (!screen) return;
const projected = screenPointInViewer(point, viewer, screen);
cursor.style.transform = `translate3d(${projected.x}px, ${projected.y}px, 0)`;
cursor.dataset.action = action ?? "move";
const pressing = action === "click" || action === "double_click" || action === "right_click";
const pulse = action === "double_click" ? (progress * 2) % 1 : progress;
ripple.style.opacity = pressing && !reducedMotion ? String(0.55 * (1 - pulse)) : "0";
ripple.style.transform = `scale(${0.35 + pulse * 1.15})`;
cursor.style.setProperty("--agent-cursor-press", pressing && pulse < 0.35 ? "0.86" : "1");
const nextTyping = action === "type" || action === "key";
if (nextTyping !== shownTyping) {
shownTyping = nextTyping;
setTyping(nextTyping);
}
};
const rest = () => {
clearTimeout(idleTimer);
idleTimer = setTimeout(() => {
cursor.style.opacity = "0";
}, POINTER_IDLE_MS);
};
const tick = () => {
frame = 0;
const now = performance.now();
const state = pointerFrameAt(segments, now, reducedMotion);
if (!state) return;
position = state.point;
place(state.point, state.action?.kind ?? null, state.action?.progress ?? 0);
if (state.done) {
segments = [];
rest();
return;
}
frame = requestAnimationFrame(tick);
};
const unsubscribe = agentPointerFeed.subscribe(botId, (payload) => {
const now = performance.now();
const current = pointerFrameAt(segments, now, reducedMotion)?.point ?? position;
const sameScreen =
screen?.width === payload.screen.width && screen?.height === payload.screen.height;
screen = payload.screen;
// A first announcement starts at its target rather than flying in from a corner.
const firstTarget = payload.steps.find((step) => typeof step.x === "number");
const from =
current && sameScreen
? current
: firstTarget
? { x: firstTarget.x ?? 0, y: firstTarget.y ?? 0 }
: { x: payload.screen.width / 2, y: payload.screen.height / 2 };
segments = planPointerTimeline(from, payload.steps, now);
clearTimeout(idleTimer);
cursor.style.opacity = "1";
if (!frame) frame = requestAnimationFrame(tick);
});
const resize = new ResizeObserver(() => {
viewer = { width: layer.clientWidth, height: layer.clientHeight };
if (position && !frame) place(position, null, 0);
});
resize.observe(layer);
return () => {
unsubscribe();
resize.disconnect();
cancelAnimationFrame(frame);
clearTimeout(idleTimer);
};
}, [botId]);
return (
<div
ref={layerRef}
aria-hidden
data-testid="agent-cursor-layer"
className="pointer-events-none absolute inset-0 overflow-hidden"
>
<div
ref={cursorRef}
data-testid="agent-cursor"
className="absolute top-0 left-0 opacity-0 transition-opacity duration-300 will-change-transform"
style={{ color }}
>
<span
ref={rippleRef}
className="absolute -top-5 -left-5 size-10 rounded-full border-2 border-current opacity-0"
/>
<svg
viewBox="0 0 20 22"
className="relative block size-5 origin-top-left drop-shadow-md transition-transform duration-100"
style={{ transform: "scale(var(--agent-cursor-press, 1))" }}
>
<path
d="M1.5 1.5 L1.5 17.5 L6 13.2 L9.2 20.2 L12.4 18.8 L9.3 12 L15.5 12 Z"
fill="currentColor"
className="stroke-background"
strokeWidth="1.6"
strokeLinejoin="round"
/>
</svg>
<span className="absolute top-5 left-3.5 flex max-w-40 items-center gap-1 rounded-full bg-popover px-2 py-0.5 text-ui-label font-medium whitespace-nowrap text-popover-foreground shadow-sm ring-1 ring-border">
<span className="size-1.5 shrink-0 rounded-full bg-current" style={{ color }} />
<span className="truncate">{name}</span>
{typing ? <Keyboard strokeWidth={1.8} className="size-3 shrink-0" /> : null}
</span>
</div>
</div>
);
}

View File

@ -0,0 +1,214 @@
import type { Edl } from "@rakazo/contracts";
import { cn } from "@rakazo/ui-web";
import { useCallback, useEffect, useImperativeHandle, useRef, useState } from "react";
import { assetFileUrl, useMediaObjectUrl } from "./media-files";
import { clipLengthMs, clipOutputStartMs, edlDurationMs, locateOutputTime } from "./timeline";
export interface PlayerHandle {
/** Jump to an output timestamp in ms. */
seek(ms: number): void;
}
/** What the source player currently shows, in source coordinates. */
export interface SourceFrame {
outputMs: number;
clipId: string;
assetId: string;
sourceMs: number;
}
/** Plays a finished render. Reports the playhead in output ms. */
export function RenderVideo({
src,
poster,
handleRef,
onTime,
className,
label,
}: {
src: string | null;
poster?: string | null;
handleRef?: React.Ref<PlayerHandle>;
onTime?: (ms: number) => void;
className?: string;
label: string;
}) {
const video = useRef<HTMLVideoElement>(null);
const pending = useRef<number | null>(null);
useImperativeHandle(handleRef, () => ({
seek(ms) {
const element = video.current;
if (!element || element.readyState < 1) {
pending.current = ms;
return;
}
element.currentTime = ms / 1000;
onTime?.(ms);
},
}));
return (
// biome-ignore lint/a11y/useMediaCaption: renders carry burned-in or sidecar captions.
<video
ref={video}
aria-label={label}
className={cn("aspect-video w-full rounded-lg bg-muted object-contain", className)}
controls
playsInline
preload="metadata"
poster={poster ?? undefined}
src={src ?? undefined}
onLoadedMetadata={(event) => {
if (pending.current === null) return;
event.currentTarget.currentTime = pending.current / 1000;
pending.current = null;
}}
onTimeUpdate={(event) => onTime?.(Math.round(event.currentTarget.currentTime * 1000))}
/>
);
}
/**
* Plays a plan's timeline from the source proxies (the unredacted "before"
* view, and the only view of a render the gate blocked). Clips play in order;
* gaps and the intro show nothing.
*/
export function SourcePlayer({
edl,
handleRef,
onFrame,
label,
children,
}: {
edl: Edl;
handleRef?: React.Ref<PlayerHandle>;
onFrame?: (frame: SourceFrame | null) => void;
label: string;
/** Overlays positioned over the video frame (masks, mark prompts). */
children?: React.ReactNode;
}) {
const [outputMs, setOutputMs] = useState(() => clipOutputStartMs(edl, firstClip(edl)));
const located = locateOutputTime(edl, outputMs);
const assetId = located?.clip.assetId ?? null;
const src = useMediaObjectUrl(assetId ? assetFileUrl(assetId, "proxy") : null);
const video = useRef<HTMLVideoElement>(null);
const playing = useRef(false);
const [ratio, setRatio] = useState(16 / 9);
const report = useCallback(
(ms: number) => {
const at = locateOutputTime(edl, ms);
onFrame?.(
at
? { outputMs: ms, clipId: at.clip.id, assetId: at.clip.assetId, sourceMs: at.sourceMs }
: null,
);
},
[edl, onFrame],
);
const seek = useCallback(
(ms: number) => {
const clamped = Math.max(0, Math.min(ms, edlDurationMs(edl)));
setOutputMs(clamped);
const at = locateOutputTime(edl, clamped);
const element = video.current;
if (at && element && at.clip.assetId === assetId && element.readyState >= 1) {
element.currentTime = at.sourceMs / 1000;
}
report(clamped);
},
[edl, assetId, report],
);
useImperativeHandle(handleRef, () => ({ seek }), [seek]);
// A new clip's proxy loaded: start at the mapped source time.
const onLoaded = (element: HTMLVideoElement) => {
if (element.videoWidth > 0 && element.videoHeight > 0) {
setRatio(element.videoWidth / element.videoHeight);
}
if (located) element.currentTime = located.sourceMs / 1000;
if (playing.current) void element.play().catch(() => undefined);
};
const onTimeUpdate = (element: HTMLVideoElement) => {
if (!located) return;
const { clip } = located;
const sourceMs = element.currentTime * 1000;
const clipEnd = clipOutputStartMs(edl, clip) + clipLengthMs(clip);
if (sourceMs >= clip.outMs) {
const next = nextClipStart(edl, clipEnd);
if (next === null) {
element.pause();
playing.current = false;
return;
}
setOutputMs(next);
report(next);
return;
}
const ms = clipOutputStartMs(edl, clip) + Math.round((sourceMs - clip.inMs) / clip.speed);
setOutputMs(ms);
report(ms);
};
// Report once per plan; playback reports through onTimeUpdate.
useEffect(() => {
report(outputMs);
}, [edl]);
return (
<div
className="relative aspect-video w-full overflow-hidden rounded-lg bg-muted"
data-testid="media-source-player"
data-position-ms={outputMs}
>
{/* biome-ignore lint/a11y/useMediaCaption: source footage has no caption track. */}
<video
ref={video}
aria-label={label}
className="absolute inset-0 h-full w-full object-contain"
controls
playsInline
preload="auto"
src={src ?? undefined}
onLoadedMetadata={(event) => onLoaded(event.currentTarget)}
onPlay={() => {
playing.current = true;
}}
onPause={() => {
playing.current = false;
}}
onTimeUpdate={(event) => onTimeUpdate(event.currentTarget)}
/>
{children ? (
// Children get the displayed frame, so normalized boxes map onto the picture.
<div className="pointer-events-none absolute inset-0 flex items-center justify-center">
<div
data-frame
className="relative"
style={{
aspectRatio: ratio,
...(ratio >= 16 / 9 ? { width: "100%" } : { height: "100%" }),
}}
>
{children}
</div>
</div>
) : null}
</div>
);
}
function firstClip(edl: Edl) {
return [...edl.tracks.video].sort((a, b) => a.atMs - b.atMs)[0] ?? edl.tracks.video[0]!;
}
/** Output start of the first clip beginning at or after `ms`, or null. */
function nextClipStart(edl: Edl, ms: number): number | null {
let best: number | null = null;
for (const clip of edl.tracks.video) {
const start = clipOutputStartMs(edl, clip);
if (start >= ms && (best === null || start < best)) best = start;
}
return best;
}

View File

@ -0,0 +1,191 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
MarketingCopy,
MediaPublication,
MediaRender,
MediaRenderOutput,
} from "@rakazo/contracts";
import { Button, cn, Input, Textarea } from "@rakazo/ui-web";
import { Download } from "lucide-react";
import { useState } from "react";
import { rpc } from "../../../lib/rpc";
import { LoadingState } from "../../ai/primitives";
import { ErrorLine, errorMessage } from "./chrome";
import {
bundleUrl,
downloadMediaFile,
mediaArtifactUrl,
renderFileUrl,
useMediaObjectUrl,
} from "./media-files";
import { thumbnailVariant } from "./renders";
/**
* Thumbnails, copy, and the publication bundle for a final render that passed
* its gate. Each step appears once the one before it has produced something.
*/
export function PublishPanel({ render }: { render: MediaRender }) {
const { t, i18n } = useLingui();
const [thumbs, setThumbs] = useState<MediaRenderOutput[]>(() =>
render.outputs.filter((output) => output.kind === "thumbnail"),
);
const [chosen, setChosen] = useState<string | null>(null);
const [copy, setCopy] = useState<MarketingCopy | null>(null);
const [title, setTitle] = useState("");
const [caption, setCaption] = useState("");
const [tags, setTags] = useState("");
const [publication, setPublication] = useState<MediaPublication | null>(null);
const [busy, setBusy] = useState<"thumbs" | "copy" | "bundle" | null>(null);
const [error, setError] = useState<string | null>(null);
async function run(step: "thumbs" | "copy" | "bundle", work: () => Promise<void>) {
setBusy(step);
setError(null);
try {
await work();
} catch (cause) {
setError(errorMessage(cause));
} finally {
setBusy(null);
}
}
const makeThumbs = () =>
run("thumbs", async () => {
const outputs = await rpc.media.renders.thumbnails({ renderId: render.id, count: 3 });
setThumbs(outputs);
setChosen(outputs[0] ? thumbnailVariant(outputs[0]) : null);
});
const makeCopy = () =>
run("copy", async () => {
const next = await rpc.media.marketing.copy({
renderId: render.id,
language: i18n.locale.slice(0, 2) || "en",
});
setCopy(next);
setTitle(next.title);
setCaption([next.hook, next.caption].filter(Boolean).join("\n\n"));
setTags(next.hashtags.join(" "));
});
const download = (bundled: MediaPublication) =>
downloadMediaFile(
bundled.bundleArtifactId ? mediaArtifactUrl(bundled.bundleArtifactId) : bundleUrl(bundled.id),
`bundle-${bundled.id}.zip`,
);
const makeBundle = () =>
run("bundle", async () => {
if (publication) {
await download(publication);
return;
}
const created = await rpc.media.publications.create({
renderId: render.id,
title: title.trim() || null,
caption: caption.trim() || null,
tags: tags
.split(/\s+/)
.map((tag) => tag.trim())
.filter(Boolean),
thumbnailVariant: chosen,
});
const bundled = await rpc.media.publications.bundle({ publicationId: created.id });
setPublication(bundled);
await download(bundled);
});
return (
<section className="flex flex-col gap-3" data-testid="media-publish">
{thumbs.length === 0 ? (
<Button variant="outline" size="sm" disabled={busy !== null} onClick={makeThumbs}>
{busy === "thumbs" ? <LoadingState label={t`Thumbnails`} /> : <Trans>Thumbnails</Trans>}
</Button>
) : (
<div className="grid grid-cols-3 gap-2">
{thumbs.map((output) => {
const variant = thumbnailVariant(output);
return (
<Thumbnail
key={output.fileName}
src={renderFileUrl(render.id, output.fileName)}
selected={chosen === variant}
label={variant}
onSelect={() => setChosen(variant)}
/>
);
})}
</div>
)}
{copy ? (
<div className="flex flex-col gap-2">
<Input aria-label={t`Title`} value={title} onChange={(e) => setTitle(e.target.value)} />
<Textarea
aria-label={t`Caption`}
rows={4}
value={caption}
onChange={(e) => setCaption(e.target.value)}
/>
<Input aria-label={t`Hashtags`} value={tags} onChange={(e) => setTags(e.target.value)} />
{copy.issues.map((issue) => (
<p
key={`${issue.code}-${issue.message}`}
className={cn(
"text-ui-body-sm",
issue.severity === "block" ? "text-destructive" : "text-warning",
)}
>
{issue.message}
</p>
))}
</div>
) : (
<Button variant="outline" size="sm" disabled={busy !== null} onClick={makeCopy}>
{busy === "copy" ? <LoadingState label={t`Copy`} /> : <Trans>Write copy</Trans>}
</Button>
)}
<Button size="sm" disabled={busy !== null} onClick={makeBundle}>
{busy === "bundle" ? (
<LoadingState label={t`Bundle`} />
) : (
<>
<Download />
{publication ? <Trans>Download again</Trans> : <Trans>Bundle</Trans>}
</>
)}
</Button>
<ErrorLine error={error} />
</section>
);
}
function Thumbnail({
src,
selected,
label,
onSelect,
}: {
src: string;
selected: boolean;
label: string;
onSelect: () => void;
}) {
const url = useMediaObjectUrl(src);
return (
<button
type="button"
aria-pressed={selected}
aria-label={label}
onClick={onSelect}
className={cn(
"aspect-video overflow-hidden rounded-md border bg-muted",
selected ? "border-primary ring-2 ring-ring/50" : "border-border",
)}
>
{url ? <img src={url} alt="" className="h-full w-full object-cover" /> : null}
</button>
);
}

View File

@ -0,0 +1,321 @@
import type { DragEndEvent, DragMoveEvent } from "@dnd-kit/core";
import {
DndContext,
KeyboardSensor,
PointerSensor,
useDraggable,
useSensor,
useSensors,
} from "@dnd-kit/core";
import { useLingui } from "@lingui/react/macro";
import type { Edl } from "@rakazo/contracts";
import { cn } from "@rakazo/ui-web";
import { useEffect, useRef, useState } from "react";
import type { DragId } from "./timeline";
import {
applyDrag,
clipLengthMs,
edlDurationMs,
formatMs,
introMs,
mainProgramMs,
} from "./timeline";
export type TimelineSelection =
| { kind: "clip"; id: string }
| { kind: "overlay"; id: string }
| { kind: "captions" }
| null;
/**
* Tracks of a plan: video clips (move, trim), overlays (move), audio, and
* captions. Positions are output time; the intro occupies the start.
*/
export function Timeline({
edl,
assetDurations,
selection,
invalid,
playheadMs,
onSelect,
onSeek,
onChange,
}: {
edl: Edl;
assetDurations: ReadonlyMap<string, number | null>;
selection: TimelineSelection;
/** Clip and overlay ids the last save rejected. */
invalid: ReadonlySet<string>;
playheadMs: number;
onSelect: (selection: TimelineSelection) => void;
onSeek: (ms: number) => void;
onChange: (edl: Edl) => void;
}) {
const { t } = useLingui();
const container = useRef<HTMLDivElement>(null);
const [width, setWidth] = useState(800);
const [drag, setDrag] = useState<{ id: DragId; deltaMs: number } | null>(null);
const sensors = useSensors(
useSensor(PointerSensor, { activationConstraint: { distance: 3 } }),
useSensor(KeyboardSensor),
);
useEffect(() => {
const element = container.current;
if (!element) return;
const observer = new ResizeObserver(([entry]) => {
if (entry) setWidth(entry.contentRect.width);
});
observer.observe(element);
return () => observer.disconnect();
}, []);
const shown = drag ? applyDrag(edl, drag.id, drag.deltaMs, assetDurations) : edl;
const totalMs = Math.max(5000, Math.round(edlDurationMs(edl) * 1.15));
const pxPerMs = Math.max(0.005, width / totalMs);
const offset = introMs(shown);
const x = (ms: number) => ms * pxPerMs;
const toMs = (event: DragMoveEvent | DragEndEvent) => Math.round(event.delta.x / pxPerMs);
return (
<DndContext
sensors={sensors}
onDragMove={(event) => setDrag({ id: event.active.id as DragId, deltaMs: toMs(event) })}
onDragCancel={() => setDrag(null)}
onDragEnd={(event) => {
setDrag(null);
const deltaMs = toMs(event);
if (deltaMs !== 0) {
onChange(applyDrag(edl, event.active.id as DragId, deltaMs, assetDurations));
}
}}
>
<div
ref={container}
className="relative flex select-none flex-col gap-1 overflow-hidden rounded-lg border border-border bg-card p-2"
data-testid="media-timeline"
>
<button
type="button"
aria-label={t`Timeline position`}
className="relative h-5 w-full cursor-pointer text-left text-ui-caption text-muted-foreground"
onClick={(event) => {
const box = event.currentTarget.getBoundingClientRect();
onSeek(Math.round((event.clientX - box.left) / pxPerMs));
}}
>
{ticks(totalMs).map((ms) => (
<span key={ms} className="absolute top-0" style={{ left: x(ms) }}>
{formatMs(ms)}
</span>
))}
</button>
<Track label={t`Video`}>
{shown.intro ? (
<div
className="absolute inset-y-0 rounded-md bg-muted"
style={{ left: 0, width: x(shown.intro.durationMs) }}
/>
) : null}
{shown.tracks.video.map((clip) => (
<ClipBlock
key={clip.id}
id={clip.id}
left={x(offset + clip.atMs)}
width={x(clipLengthMs(clip))}
selected={selection?.kind === "clip" && selection.id === clip.id}
invalid={invalid.has(clip.id)}
label={t`Clip ${clip.id}`}
onSelect={() => onSelect({ kind: "clip", id: clip.id })}
/>
))}
</Track>
<Track label={t`Overlays`}>
{shown.overlays.map((overlay) => (
<OverlayBlock
key={overlay.id}
id={overlay.id}
left={x(offset + overlay.atMs)}
width={x(overlay.durationMs)}
text={overlay.text ?? overlay.kind}
selected={selection?.kind === "overlay" && selection.id === overlay.id}
invalid={invalid.has(overlay.id)}
onSelect={() => onSelect({ kind: "overlay", id: overlay.id })}
/>
))}
</Track>
{shown.tracks.audio.length > 0 ? (
<Track label={t`Audio`}>
{shown.tracks.audio.map((clip) => (
<div
key={clip.id}
className="absolute inset-y-1 rounded-md bg-secondary"
style={{
left: x(offset + clip.atMs),
width: x(Math.max(0, clip.outMs - clip.inMs)),
}}
/>
))}
</Track>
) : null}
<Track label={t`Captions`}>
<button
type="button"
aria-pressed={selection?.kind === "captions"}
onClick={() => onSelect({ kind: "captions" })}
className={cn(
"absolute inset-y-1 rounded-md border text-left text-ui-caption",
shown.captions && shown.captions.mode !== "none"
? "border-border bg-accent px-2 text-accent-foreground"
: "border-dashed border-border px-2 text-muted-foreground",
selection?.kind === "captions" && "ring-2 ring-ring",
)}
style={{ left: x(offset), width: Math.max(x(mainProgramMs(shown)), 48) }}
>
{t`Captions`}
</button>
</Track>
<div
aria-hidden
className="pointer-events-none absolute top-2 bottom-2 w-px bg-primary"
style={{ left: 8 + x(playheadMs) }}
/>
</div>
</DndContext>
);
}
function Track({ label, children }: { label: string; children: React.ReactNode }) {
return (
// biome-ignore lint/a11y/useSemanticElements: a timeline track is a labelled group of blocks.
<div role="group" aria-label={label} className="relative h-10 rounded-md bg-muted/40">
{children}
</div>
);
}
function ClipBlock({
id,
left,
width,
selected,
invalid,
label,
onSelect,
}: {
id: string;
left: number;
width: number;
selected: boolean;
invalid: boolean;
label: string;
onSelect: () => void;
}) {
const { t } = useLingui();
const move = useDraggable({ id: `clip:${id}` });
const start = useDraggable({ id: `trim-start:${id}` });
const end = useDraggable({ id: `trim-end:${id}` });
const handle = "h-full w-2 shrink-0 cursor-ew-resize bg-primary/40 focus-visible:bg-primary";
return (
<div
data-testid="media-clip"
data-clip-id={id}
data-invalid={invalid || undefined}
className={cn(
"absolute inset-y-1 flex items-stretch overflow-hidden rounded-md border bg-primary/15 text-ui-caption text-foreground",
selected ? "border-primary ring-2 ring-ring/50" : "border-primary/30",
invalid && "border-destructive ring-2 ring-destructive/40",
move.isDragging && "opacity-80",
)}
style={{ left, width: Math.max(width, 8) }}
>
<button
type="button"
ref={start.setNodeRef}
{...start.attributes}
{...start.listeners}
aria-label={t`Trim start`}
data-testid="media-trim-start"
className={handle}
/>
<button
type="button"
ref={move.setNodeRef}
{...move.attributes}
{...move.listeners}
aria-label={label}
aria-pressed={selected}
onClick={onSelect}
className={cn(
"min-w-0 flex-1 cursor-grab truncate px-1 text-left",
move.isDragging && "cursor-grabbing",
)}
>
{id}
</button>
<button
type="button"
ref={end.setNodeRef}
{...end.attributes}
{...end.listeners}
aria-label={t`Trim end`}
data-testid="media-trim-end"
className={handle}
/>
</div>
);
}
function OverlayBlock({
id,
left,
width,
text,
selected,
invalid,
onSelect,
}: {
id: string;
left: number;
width: number;
text: string;
selected: boolean;
invalid: boolean;
onSelect: () => void;
}) {
const move = useDraggable({ id: `overlay:${id}` });
return (
<button
type="button"
ref={move.setNodeRef}
{...move.attributes}
{...move.listeners}
aria-label={text}
aria-pressed={selected}
data-testid="media-overlay"
onClick={onSelect}
className={cn(
"absolute inset-y-1 cursor-grab truncate rounded-md border bg-secondary px-1 text-left text-ui-caption text-secondary-foreground",
selected ? "border-primary ring-2 ring-ring/50" : "border-border",
invalid && "border-destructive ring-2 ring-destructive/40",
)}
style={{ left, width: Math.max(width, 8) }}
>
{text}
</button>
);
}
/** Ruler marks every 1, 5, 10, or 30 seconds, whichever keeps about eight labels. */
function ticks(totalMs: number): number[] {
const step = [1000, 5000, 10_000, 30_000, 60_000].find((ms) => totalMs / ms <= 10) ?? 120_000;
const out: number[] = [];
for (let ms = 0; ms < totalMs; ms += step) out.push(ms);
return out;
}

View File

@ -0,0 +1,60 @@
import { useLingui } from "@lingui/react/macro";
import { Badge, Button, cn } from "@rakazo/ui-web";
import { ChevronLeft } from "lucide-react";
import type { GateStatus } from "./renders";
/** Header row of an editor-side view: back, title, and the view's actions. */
export function ViewHeader({
title,
onBack,
backLabel,
children,
}: {
title: React.ReactNode;
onBack: () => void;
backLabel: string;
children?: React.ReactNode;
}) {
return (
<header className="flex min-h-control-lg items-center gap-2">
<Button variant="ghost" size="icon-sm" aria-label={backLabel} onClick={onBack}>
<ChevronLeft />
</Button>
<h2 className="min-w-0 flex-1 truncate text-ui-title font-semibold">{title}</h2>
{children}
</header>
);
}
/** Gate outcome as a small status badge. */
export function GateBadge({ status }: { status: GateStatus }) {
const { t } = useLingui();
const label = { pass: t`Passed`, warn: t`Warnings`, blocked: t`Blocked` }[status];
return (
<Badge
data-testid="media-gate-status"
data-status={status}
variant={status === "blocked" ? "destructive" : "outline"}
className={cn(
status === "pass" && "border-success/40 text-success",
status === "warn" && "border-warning/40 text-warning",
)}
>
{label}
</Badge>
);
}
/** A request error in one line, under the control that caused it. */
export function ErrorLine({ error }: { error: string | null }) {
if (!error) return null;
return (
<p role="alert" className="text-ui-body-sm text-destructive">
{error}
</p>
);
}
export function errorMessage(error: unknown): string {
return error instanceof Error && error.message ? error.message : String(error);
}

View File

@ -0,0 +1,147 @@
import { Field, FieldLabel, Input, NativeSelect, NativeSelectOption, Switch } from "@rakazo/ui-web";
import { useEffect, useId, useState } from "react";
/*
* Compact labelled inputs for the inspector panels. Times are edited in
* seconds and stored in milliseconds.
*/
export function SecondsField({
label,
ms,
min = 0,
onChange,
}: {
label: string;
ms: number;
min?: number;
onChange: (ms: number) => void;
}) {
const id = useId();
const [text, setText] = useState(() => (ms / 1000).toFixed(1));
useEffect(() => setText((ms / 1000).toFixed(1)), [ms]);
return (
<Field className="gap-1">
<FieldLabel htmlFor={id}>{label}</FieldLabel>
<Input
id={id}
type="number"
inputMode="decimal"
step={0.1}
min={min / 1000}
value={text}
onChange={(event) => setText(event.target.value)}
onBlur={() => {
const seconds = Number(text);
if (Number.isFinite(seconds) && seconds * 1000 >= min)
onChange(Math.round(seconds * 1000));
else setText((ms / 1000).toFixed(1));
}}
/>
</Field>
);
}
export function NumberField({
label,
value,
min,
max,
step = 1,
onChange,
}: {
label: string;
value: number;
min: number;
max: number;
step?: number;
onChange: (value: number) => void;
}) {
const id = useId();
return (
<Field className="gap-1">
<FieldLabel htmlFor={id}>{label}</FieldLabel>
<Input
id={id}
type="number"
min={min}
max={max}
step={step}
value={value}
onChange={(event) => {
const next = Number(event.target.value);
if (Number.isFinite(next)) onChange(Math.min(max, Math.max(min, next)));
}}
/>
</Field>
);
}
export function SelectField<T extends string>({
label,
value,
options,
onChange,
}: {
label: string;
value: T;
options: ReadonlyArray<{ value: T; label: string }>;
onChange: (value: T) => void;
}) {
const id = useId();
return (
<Field className="gap-1">
<FieldLabel htmlFor={id}>{label}</FieldLabel>
<NativeSelect id={id} value={value} onChange={(event) => onChange(event.target.value as T)}>
{options.map((option) => (
<NativeSelectOption key={option.value} value={option.value}>
{option.label}
</NativeSelectOption>
))}
</NativeSelect>
</Field>
);
}
export function TextField({
label,
value,
maxLength,
onChange,
}: {
label: string;
value: string;
maxLength?: number;
onChange: (value: string) => void;
}) {
const id = useId();
return (
<Field className="gap-1">
<FieldLabel htmlFor={id}>{label}</FieldLabel>
<Input
id={id}
value={value}
maxLength={maxLength}
onChange={(event) => onChange(event.target.value)}
/>
</Field>
);
}
export function ToggleField({
label,
checked,
onChange,
}: {
label: string;
checked: boolean;
onChange: (checked: boolean) => void;
}) {
const id = useId();
return (
<Field orientation="horizontal" className="items-center gap-2">
<Switch id={id} checked={checked} onCheckedChange={onChange} />
<FieldLabel htmlFor={id}>{label}</FieldLabel>
</Field>
);
}

View File

@ -0,0 +1,93 @@
import type { MediaRender } from "@rakazo/contracts";
import { useEffect, useRef, useState } from "react";
import { withSpaceHeaders } from "../../../lib/rpc";
/*
* Media byte routes need the selected space header, which `<video src>` and
* `<img src>` cannot send. Files are fetched with the header and shown through
* object URLs; renders and previews are short, so whole-file fetches are fine.
*/
export function renderFileUrl(renderId: string, fileName: string, download = false): string {
const base = `/api/media/renders/${encodeURIComponent(renderId)}/files/${encodeURIComponent(fileName)}`;
return download ? `${base}?download=1` : base;
}
export function assetFileUrl(assetId: string, kind: "thumb" | "proxy"): string {
return `/api/media/assets/${encodeURIComponent(assetId)}/${kind}`;
}
/** Thumbnails, intro previews, and other media outputs stored as space artifacts. */
export function mediaArtifactUrl(artifactId: string): string {
return `/api/media/artifacts/${encodeURIComponent(artifactId)}`;
}
export function bundleUrl(publicationId: string): string {
return `/api/media/bundles/${encodeURIComponent(publicationId)}`;
}
/** The streamable video of a render, when it has one. */
export function renderVideoUrl(render: Pick<MediaRender, "id" | "outputs">): string | null {
const video = render.outputs.find((output) => output.kind === "video");
return video ? renderFileUrl(render.id, video.fileName) : null;
}
export function renderPosterUrl(render: Pick<MediaRender, "id" | "outputs">): string | null {
const poster = render.outputs.find((output) => output.kind === "poster");
return poster ? renderFileUrl(render.id, poster.fileName) : null;
}
/** An object URL for a media byte route, or null while loading or when missing. */
export function useMediaObjectUrl(url: string | null): string | null {
const [objectUrl, setObjectUrl] = useState<string | null>(null);
useEffect(() => {
setObjectUrl(null);
if (!url) return;
const controller = new AbortController();
let created: string | null = null;
void fetch(url, {
headers: withSpaceHeaders(),
credentials: "include",
signal: controller.signal,
})
.then(async (response) => {
if (!response.ok) return;
const blob = await response.blob();
if (controller.signal.aborted) return;
created = URL.createObjectURL(blob);
setObjectUrl(created);
})
.catch(() => undefined);
return () => {
controller.abort();
if (created) URL.revokeObjectURL(created);
};
}, [url]);
return objectUrl;
}
/** Download a media byte route as a file. */
export async function downloadMediaFile(url: string, fileName: string): Promise<void> {
const response = await fetch(url, { headers: withSpaceHeaders(), credentials: "include" });
if (!response.ok) throw new Error(`Download failed (${response.status})`);
const href = URL.createObjectURL(await response.blob());
const anchor = document.createElement("a");
anchor.href = href;
anchor.download = fileName;
anchor.click();
window.setTimeout(() => URL.revokeObjectURL(href), 60_000);
}
/**
* Run `tick` every `intervalMs` while `active`; stops as soon as work is idle.
* The latest `tick` is always used, so callers need not memoize it.
*/
export function usePolling(tick: () => unknown, active: boolean, intervalMs: number): void {
const latest = useRef(tick);
latest.current = tick;
useEffect(() => {
if (!active) return;
const timer = window.setInterval(() => void latest.current(), intervalMs);
return () => window.clearInterval(timer);
}, [active, intervalMs]);
}

View File

@ -0,0 +1,135 @@
import type { GateIssue, MediaEditPlan, MediaRender } from "@rakazo/contracts";
import { describe, expect, it } from "vitest";
import {
gateStatus,
projectVariants,
reviewedPlan,
sortedIssues,
thumbnailVariant,
} from "./renders";
function render(patch: Partial<MediaRender>): MediaRender {
return {
id: "r1",
planId: "p1",
projectId: "proj",
profileId: "youtube-shorts",
quality: "draft",
status: "done",
progress: 1,
outputs: [],
gateReport: null,
ffprobe: null,
error: null,
createdAt: "2026-10-01T10:00:00.000Z",
updatedAt: "2026-10-01T10:00:00.000Z",
...patch,
};
}
function plan(patch: Partial<MediaEditPlan>): MediaEditPlan {
return {
id: "p1",
projectId: "proj",
profileId: "youtube-shorts",
version: 1,
edl: {} as MediaEditPlan["edl"],
createdBy: "user",
rationale: null,
createdAt: "2026-10-01T09:00:00.000Z",
...patch,
};
}
const issue = (patch: Partial<GateIssue>): GateIssue => ({
check: "content",
severity: "block",
code: "x",
message: "m",
...patch,
});
const report = (issues: GateIssue[], passed: boolean, autoFixRound = 0) => ({
profileId: "youtube-shorts",
passed,
checkedAt: "2026-10-01T10:00:00.000Z",
sampledFrames: 10,
issues,
autoFixRound,
});
describe("gateStatus", () => {
it("is null while running or without a report", () => {
expect(gateStatus(render({ status: "rendering" }))).toBeNull();
expect(gateStatus(render({ gateReport: null }))).toBeNull();
});
it("separates pass, warnings, and blocked", () => {
expect(gateStatus(render({ gateReport: report([], true) }))).toBe("pass");
expect(gateStatus(render({ gateReport: report([issue({ severity: "warn" })], true) }))).toBe(
"warn",
);
expect(gateStatus(render({ status: "blocked", gateReport: report([issue({})], false) }))).toBe(
"blocked",
);
// A failed report never reads as passed, whatever the issue list says.
expect(gateStatus(render({ gateReport: report([], false) }))).toBe("blocked");
});
});
describe("sortedIssues", () => {
it("orders by time, untimed last, blockers first at equal time", () => {
const sorted = sortedIssues([
issue({ code: "untimed" }),
issue({ code: "late", atMs: 5000 }),
issue({ code: "warn", atMs: 1000, severity: "warn" }),
issue({ code: "block", atMs: 1000 }),
]);
expect(sorted.map((item) => item.code)).toEqual(["block", "warn", "late", "untimed"]);
});
});
describe("projectVariants", () => {
it("keeps one card per profile with the newest plan and renders", () => {
const plans = [
plan({ id: "p1", version: 1 }),
plan({ id: "p2", version: 2 }),
plan({ id: "q1", profileId: "tiktok" }),
];
const renders = [
render({ id: "d1", createdAt: "2026-10-01T10:00:00.000Z" }),
render({ id: "d2", createdAt: "2026-10-01T11:00:00.000Z" }),
render({ id: "f1", quality: "final" }),
];
const variants = projectVariants(["youtube-shorts", "instagram-reels"], plans, renders);
expect(variants.map((variant) => variant.profileId)).toEqual([
"youtube-shorts",
"instagram-reels",
"tiktok",
]);
expect(variants[0]?.plan?.id).toBe("p2");
expect(variants[0]?.draft?.id).toBe("d2");
expect(variants[0]?.final?.id).toBe("f1");
expect(variants[1]?.plan).toBeNull();
});
});
describe("reviewedPlan", () => {
it("follows auto-fix versions created by the gate", () => {
const plans = [
plan({ id: "p1", version: 1 }),
plan({ id: "p2", version: 2, createdBy: "agent" }),
plan({ id: "p3", version: 3, createdBy: "agent" }),
];
expect(reviewedPlan(render({ planId: "p1" }), plans)?.id).toBe("p1");
expect(
reviewedPlan(render({ planId: "p1", gateReport: report([], false, 1) }), plans)?.id,
).toBe("p2");
});
});
describe("thumbnailVariant", () => {
it("strips the extension", () => {
expect(thumbnailVariant({ fileName: "thumb-b.png" })).toBe("thumb-b");
});
});

View File

@ -0,0 +1,108 @@
import type {
GateIssue,
MediaEditPlan,
MediaRender,
MediaRenderOutput,
MediaRenderQuality,
MediaRenderStatus,
} from "@rakazo/contracts";
/** Renders still moving; clients poll while any is in one of these states. */
const ACTIVE: ReadonlySet<MediaRenderStatus> = new Set(["queued", "rendering", "gating"]);
/** How often to re-read a render (or project) while work is in progress. */
export const RENDER_POLL_MS = 2000;
export function isRenderActive(render: Pick<MediaRender, "status">): boolean {
return ACTIVE.has(render.status);
}
export type GateStatus = "pass" | "warn" | "blocked";
/** Gate outcome of a finished render, or null while there is no report yet. */
export function gateStatus(render: Pick<MediaRender, "status" | "gateReport">): GateStatus | null {
const report = render.gateReport;
if (render.status === "blocked") return "blocked";
if (!report || isRenderActive(render)) return null;
if (!report.passed || report.issues.some((issue) => issue.severity === "block")) {
return "blocked";
}
return report.issues.length > 0 ? "warn" : "pass";
}
/** Gate issues in timeline order (untimed ones last), blockers before warnings at equal times. */
export function sortedIssues(issues: GateIssue[]): GateIssue[] {
const at = (issue: GateIssue) => issue.atMs ?? Number.POSITIVE_INFINITY;
return [...issues].sort((a, b) => {
if (at(a) !== at(b)) return at(a) < at(b) ? -1 : 1;
if (a.severity !== b.severity) return a.severity === "block" ? -1 : 1;
return a.code.localeCompare(b.code);
});
}
function newest<T extends { createdAt: string }>(items: T[]): T | null {
let best: T | null = null;
for (const item of items) if (!best || item.createdAt > best.createdAt) best = item;
return best;
}
export interface Variant {
profileId: string;
/** Newest plan version for the profile. */
plan: MediaEditPlan | null;
/** Newest render of the profile in each quality. */
draft: MediaRender | null;
final: MediaRender | null;
}
/** One variant per project profile: its newest plan and newest draft and final renders. */
export function projectVariants(
profileIds: string[],
plans: MediaEditPlan[],
renders: MediaRender[],
): Variant[] {
const ids = [...profileIds];
for (const plan of plans) if (!ids.includes(plan.profileId)) ids.push(plan.profileId);
return ids.map((profileId) => {
const own = plans.filter((plan) => plan.profileId === profileId);
let plan: MediaEditPlan | null = null;
for (const candidate of own) if (!plan || candidate.version > plan.version) plan = candidate;
const ofQuality = (quality: MediaRenderQuality) =>
newest(
renders.filter((render) => render.profileId === profileId && render.quality === quality),
);
return { profileId, plan, draft: ofQuality("draft"), final: ofQuality("final") };
});
}
/**
* The plan whose timeline a render's gate report refers to: the started plan,
* or the newest auto-fix version of it when the gate repaired the plan.
*/
export function reviewedPlan(render: MediaRender, plans: MediaEditPlan[]): MediaEditPlan | null {
const started = plans.find((plan) => plan.id === render.planId) ?? null;
const rounds = render.gateReport?.autoFixRound ?? 0;
if (!started || rounds === 0) return started;
let best = started;
for (const plan of plans) {
if (
plan.profileId === started.profileId &&
plan.version > best.version &&
plan.version <= started.version + rounds &&
plan.createdBy === "agent"
) {
best = plan;
}
}
return best;
}
/** True when anything in a project still renders. */
export function anyRenderActive(renders: MediaRender[]): boolean {
return renders.some(isRenderActive);
}
/** Thumbnail variant name from its output file (`thumb-a.png` → `thumb-a`). */
export function thumbnailVariant(output: Pick<MediaRenderOutput, "fileName">): string {
return output.fileName.replace(/\.[a-z0-9]+$/i, "");
}

View File

@ -0,0 +1,162 @@
import { useLingui } from "@lingui/react/macro";
import type { GateIssue } from "@rakazo/contracts";
import { cn } from "@rakazo/ui-web";
import { useRef } from "react";
import type { MaskBox } from "./timeline";
import { clampBox, formatMs } from "./timeline";
/**
* A draggable, resizable mask box over the source frame. Coordinates are
* normalized to the frame; the parent must be the displayed frame itself.
*/
export function MaskBoxEditor({
box,
label,
selected,
onSelect,
onChange,
}: {
box: MaskBox;
label: string;
selected: boolean;
onSelect: () => void;
onChange: (box: MaskBox) => void;
}) {
const gesture = useRef<{
mode: "move" | "resize";
x: number;
y: number;
start: MaskBox;
frame: DOMRect;
} | null>(null);
const begin = (mode: "move" | "resize") => (event: React.PointerEvent<HTMLElement>) => {
event.stopPropagation();
event.preventDefault();
const frame = event.currentTarget.closest("[data-frame]")?.getBoundingClientRect();
if (!frame) return;
onSelect();
event.currentTarget.setPointerCapture(event.pointerId);
gesture.current = { mode, x: event.clientX, y: event.clientY, start: box, frame };
};
const move = (event: React.PointerEvent<HTMLElement>) => {
const active = gesture.current;
if (!active) return;
const dx = (event.clientX - active.x) / active.frame.width;
const dy = (event.clientY - active.y) / active.frame.height;
const { start } = active;
onChange(
clampBox(
active.mode === "move"
? { ...start, x: start.x + dx, y: start.y + dy }
: { ...start, w: start.w + dx, h: start.h + dy },
),
);
};
const end = () => {
gesture.current = null;
};
const nudge = (event: React.KeyboardEvent) => {
const step = event.shiftKey ? 0.05 : 0.01;
const delta: Record<string, [number, number]> = {
ArrowLeft: [-step, 0],
ArrowRight: [step, 0],
ArrowUp: [0, -step],
ArrowDown: [0, step],
};
const [dx, dy] = delta[event.key] ?? [0, 0];
if (dx === 0 && dy === 0) return;
event.preventDefault();
onChange(clampBox({ ...box, x: box.x + dx, y: box.y + dy }));
};
return (
<button
type="button"
aria-label={label}
aria-pressed={selected}
data-testid="media-mask"
onPointerDown={begin("move")}
onPointerMove={move}
onPointerUp={end}
onPointerCancel={end}
onKeyDown={nudge}
className={cn(
"pointer-events-auto absolute cursor-move touch-none rounded-sm border-2 bg-foreground/20 backdrop-blur-sm",
selected ? "border-primary" : "border-background",
)}
style={{
left: `${box.x * 100}%`,
top: `${box.y * 100}%`,
width: `${box.w * 100}%`,
height: `${box.h * 100}%`,
}}
>
<span
aria-hidden
onPointerDown={begin("resize")}
onPointerMove={move}
onPointerUp={end}
onPointerCancel={end}
className="absolute -right-1.5 -bottom-1.5 size-3 cursor-nwse-resize rounded-sm border border-background bg-primary"
/>
</button>
);
}
export interface CoverageRow {
key: string;
label: string;
issues: GateIssue[];
}
/**
* Per person and per mark: a bar over the program length, tinted where the
* gate found the protection incomplete. A clean row means full coverage.
*/
export function CoverageHeatmap({
rows,
durationMs,
onSeek,
}: {
rows: CoverageRow[];
durationMs: number;
onSeek: (ms: number) => void;
}) {
const { t } = useLingui();
if (rows.length === 0 || durationMs <= 0) return null;
const pct = (ms: number) => `${Math.min(100, Math.max(0, (ms / durationMs) * 100))}%`;
return (
<div className="flex flex-col gap-1.5" data-testid="media-coverage">
{rows.map((row) => (
<div key={row.key} className="grid grid-cols-[8rem_1fr] items-center gap-2">
<span className="truncate text-ui-body-sm">{row.label}</span>
<div
className={cn(
"relative h-3 rounded-full",
row.issues.length === 0 ? "bg-success/30" : "bg-muted",
)}
>
{row.issues.map((issue, index) => {
const at = issue.atMs ?? 0;
const end = Math.max(issue.endMs ?? at, at + durationMs * 0.01);
return (
<button
key={`${issue.code}-${at}-${index}`}
type="button"
aria-label={t`${formatMs(at)}: ${issue.message}`}
onClick={() => onSeek(at)}
className={cn(
"absolute inset-y-0 rounded-full",
issue.severity === "block" ? "bg-destructive" : "bg-warning",
)}
style={{ left: pct(at), width: `calc(${pct(end)} - ${pct(at)})` }}
/>
);
})}
</div>
</div>
))}
</div>
);
}

View File

@ -0,0 +1,152 @@
import type { Edl } from "@rakazo/contracts";
import { describe, expect, it } from "vitest";
import {
addRegion,
applyDrag,
clampBox,
edlDurationMs,
errorTarget,
formatMs,
freeId,
locateOutputTime,
moveClip,
regionBoxAt,
setRegionKeyframe,
trimClip,
} from "./timeline";
function edl(): Edl {
return {
edlVersion: 1,
output: {
profileId: "example-short-vertical",
width: 1080,
height: 1920,
fps: 30,
durationTargetMs: 8000,
durationHardMinMs: 1000,
durationHardMaxMs: 60000,
},
tracks: {
video: [
{
id: "a",
assetId: "asset-1",
inMs: 1000,
outMs: 4000,
atMs: 0,
speed: 1,
reframe: "auto",
stabilize: false,
muted: false,
},
{
id: "b",
assetId: "asset-2",
inMs: 0,
outMs: 4000,
atMs: 3000,
speed: 2,
reframe: "none",
stabilize: false,
muted: false,
},
],
audio: [],
},
transitions: [],
overlays: [{ id: "text-1", kind: "text", atMs: 500, durationMs: 1000, text: "Hi" }],
intro: { templateId: "intro-minimal", params: {}, durationMs: 1500 },
redactions: { faces: [], marks: [], regions: [] },
metadata: { tags: [], aiAltered: false, aiAlterations: [] },
};
}
describe("timeline math", () => {
it("counts intro, clips at their speed, and outro", () => {
// intro 1500 + clip b ends at 3000 + 4000/2
expect(edlDurationMs(edl())).toBe(6500);
});
it("maps output time to the clip and source time, intro to nothing", () => {
expect(locateOutputTime(edl(), 1000)).toBeNull();
const first = locateOutputTime(edl(), 1500 + 500);
expect(first?.clip.id).toBe("a");
expect(first?.sourceMs).toBe(1500);
const second = locateOutputTime(edl(), 1500 + 4000);
expect(second?.clip.id).toBe("b");
expect(second?.sourceMs).toBe(2000);
});
it("moves clips but never before zero", () => {
expect(moveClip(edl(), "b", 500).tracks.video[1]?.atMs).toBe(3500);
expect(moveClip(edl(), "a", -900).tracks.video[0]?.atMs).toBe(0);
});
it("trims the end inside the asset and keeps a minimum length", () => {
expect(trimClip(edl(), "a", "end", 5000, 4500).tracks.video[0]?.outMs).toBe(4500);
expect(trimClip(edl(), "a", "end", -10_000, 4500).tracks.video[0]?.outMs).toBe(1100);
});
it("trims the start without moving the clip end on the timeline", () => {
const trimmed = trimClip(edl(), "b", "start", 500, 4000).tracks.video[1];
// 500 ms of output at speed 2 is 1000 ms of source.
expect(trimmed?.inMs).toBe(1000);
expect(trimmed?.atMs).toBe(3500);
// Cannot pull the start before the asset begins.
const extended = trimClip(edl(), "a", "start", -5000, 4000).tracks.video[0];
expect(extended?.inMs).toBe(1000);
expect(extended?.atMs).toBe(0);
});
it("applies drags by handle id", () => {
const durations = new Map([
["asset-1", 4500],
["asset-2", 4000],
]);
expect(applyDrag(edl(), "trim-end:a", 300, durations).tracks.video[0]?.outMs).toBe(4300);
expect(applyDrag(edl(), "overlay:text-1", 250, durations).overlays[0]?.atMs).toBe(750);
expect(applyDrag(edl(), "clip:a", 100, durations).tracks.video[0]?.atMs).toBe(100);
});
it("points EDL errors at the clip or overlay they name", () => {
expect(errorTarget(edl(), { path: "tracks.video[1].outMs" })).toEqual({
kind: "clip",
id: "b",
});
expect(errorTarget(edl(), { path: "overlays[0].box" })).toEqual({
kind: "overlay",
id: "text-1",
});
expect(errorTarget(edl(), { path: "output" })).toBeNull();
expect(errorTarget(edl(), { path: "tracks.video[9]" })).toBeNull();
});
it("formats timestamps", () => {
expect(formatMs(0)).toBe("0:00.0");
expect(formatMs(65_300)).toBe("1:05.3");
});
it("finds free ids", () => {
expect(freeId(["mask-1", "mask-2"], "mask")).toBe("mask-3");
});
});
describe("masks", () => {
it("adds a region and keyframes it in source time", () => {
const added = addRegion(edl(), "a", 2000);
expect(added.id).toBe("mask-1");
const moved = setRegionKeyframe(added.edl, added.id, 3000, { x: 0.9, y: 0.1, w: 0.3, h: 0.2 });
const region = moved.redactions.regions[0]!;
expect(region.keyframes.map((keyframe) => keyframe.tMs)).toEqual([2000, 3000]);
// Clamped into the frame.
expect(region.keyframes[1]?.box.x).toBeCloseTo(0.7);
expect(regionBoxAt(region, 2500)).toEqual(region.keyframes[0]?.box);
expect(regionBoxAt(region, 3500)).toEqual(region.keyframes[1]?.box);
expect(regionBoxAt(region, 0)).toEqual(region.keyframes[0]?.box);
});
it("clamps boxes to a minimum size inside the frame", () => {
expect(clampBox({ x: -1, y: 2, w: 0, h: 5 })).toEqual({ x: 0, y: 0, w: 0.02, h: 1 });
});
});

View File

@ -0,0 +1,271 @@
import type { Edl, EdlError, EdlOverlay, EdlVideoClip } from "@rakazo/contracts";
/*
* Timeline math for the editor and review player. Mirrors the timing helpers
* in `@rakazo/media` (`clipLengthMs`, `edlDurationMs`, `locateOutputTime`),
* which the web app cannot import yet; the server validates every plan anyway.
*/
/** Shortest clip the editor lets a trim produce. */
export const MIN_CLIP_MS = 100;
export function clipLengthMs(clip: Pick<EdlVideoClip, "inMs" | "outMs" | "speed">): number {
return Math.max(0, Math.round((clip.outMs - clip.inMs) / clip.speed));
}
export function introMs(edl: Edl): number {
return edl.intro?.durationMs ?? 0;
}
export function mainProgramMs(edl: Edl): number {
let end = 0;
for (const clip of edl.tracks.video) end = Math.max(end, clip.atMs + clipLengthMs(clip));
return end;
}
export function edlDurationMs(edl: Edl): number {
return introMs(edl) + mainProgramMs(edl) + (edl.outro?.durationMs ?? 0);
}
/** The clip and source time an output timestamp shows; the incoming clip wins during a blend. */
export function locateOutputTime(
edl: Edl,
outputMs: number,
): { clip: EdlVideoClip; sourceMs: number } | null {
const t = outputMs - introMs(edl);
if (t < 0) return null;
const visible = edl.tracks.video
.filter((clip) => t >= clip.atMs && t < clip.atMs + clipLengthMs(clip))
.sort((a, b) => a.atMs - b.atMs || a.id.localeCompare(b.id));
const clip = visible.at(-1);
if (!clip) return null;
return {
clip,
sourceMs: Math.min(clip.outMs, Math.round(clip.inMs + (t - clip.atMs) * clip.speed)),
};
}
/** Output timestamp where a clip starts (intro included). */
export function clipOutputStartMs(edl: Edl, clip: EdlVideoClip): number {
return introMs(edl) + clip.atMs;
}
function replaceClip(edl: Edl, clipId: string, next: (clip: EdlVideoClip) => EdlVideoClip): Edl {
return {
...edl,
tracks: {
...edl.tracks,
video: edl.tracks.video.map((clip) => (clip.id === clipId ? next(clip) : clip)),
},
};
}
/** Move a clip on the output timeline. */
export function moveClip(edl: Edl, clipId: string, deltaMs: number): Edl {
return replaceClip(edl, clipId, (clip) => ({
...clip,
atMs: Math.max(0, Math.round(clip.atMs + deltaMs)),
}));
}
/**
* Trim a clip edge by an output-time delta. The start edge keeps the clip's
* end in place on the timeline; both edges stay inside the source asset.
*/
export function trimClip(
edl: Edl,
clipId: string,
edge: "start" | "end",
deltaMs: number,
assetDurationMs: number | null,
): Edl {
return replaceClip(edl, clipId, (clip) => {
const sourceDelta = Math.round(deltaMs * clip.speed);
const minSource = Math.round(MIN_CLIP_MS * clip.speed);
if (edge === "end") {
const max = assetDurationMs ?? Number.POSITIVE_INFINITY;
const outMs = Math.min(max, Math.max(clip.inMs + minSource, clip.outMs + sourceDelta));
return { ...clip, outMs };
}
// Never start before the asset, the timeline, or past the end edge.
const lowest = Math.max(0, clip.inMs - Math.round(clip.atMs * clip.speed));
const inMs = Math.min(clip.outMs - minSource, Math.max(lowest, clip.inMs + sourceDelta));
const atMs = Math.max(0, clip.atMs + Math.round((inMs - clip.inMs) / clip.speed));
return { ...clip, inMs, atMs };
});
}
/** Move an overlay on the output timeline. */
export function moveOverlay(edl: Edl, overlayId: string, deltaMs: number): Edl {
return {
...edl,
overlays: edl.overlays.map((overlay) =>
overlay.id === overlayId
? { ...overlay, atMs: Math.max(0, Math.round(overlay.atMs + deltaMs)) }
: overlay,
),
};
}
export function updateOverlay(edl: Edl, overlayId: string, patch: Partial<EdlOverlay>): Edl {
return {
...edl,
overlays: edl.overlays.map((overlay) =>
overlay.id === overlayId ? { ...overlay, ...patch } : overlay,
),
};
}
/** A fresh `prefix-n` id in `[a-z0-9_-]` that `taken` does not contain. */
export function freeId(taken: Iterable<string>, prefix: string): string {
const used = new Set(taken);
for (let index = 1; ; index += 1) {
const id = `${prefix}-${index}`;
if (!used.has(id)) return id;
}
}
/** A fresh id that no clip, overlay, or redaction of the plan uses yet. */
export function nextId(edl: Edl, prefix: string): string {
return freeId(
[
...edl.tracks.video.map((clip) => clip.id),
...edl.tracks.audio.map((clip) => clip.id),
...edl.overlays.map((overlay) => overlay.id),
...edl.redactions.faces.map((face) => face.id),
...edl.redactions.regions.map((region) => region.id),
],
prefix,
);
}
/**
* Which timeline item an EDL error points at, from its machine-readable path
* (`tracks.video[2].outMs`, `overlays[0].box`). Null for plan-wide errors.
*/
export function errorTarget(
edl: Edl,
error: Pick<EdlError, "path">,
): { kind: "clip" | "overlay"; id: string } | null {
const clip = /^tracks\.video\[(\d+)\]/.exec(error.path);
if (clip) {
const id = edl.tracks.video[Number(clip[1])]?.id;
return id ? { kind: "clip", id } : null;
}
const overlay = /^overlays\[(\d+)\]/.exec(error.path);
if (overlay) {
const id = edl.overlays[Number(overlay[1])]?.id;
return id ? { kind: "overlay", id } : null;
}
return null;
}
/** Masks drawn in review: region redactions keyed to a clip, keyframes in source time. */
export type RegionRedaction = Edl["redactions"]["regions"][number];
export type MaskBox = RegionRedaction["keyframes"][number]["box"];
/** The box a region shows at a source time: the last keyframe at or before it, else the first. */
export function regionBoxAt(region: RegionRedaction, sourceMs: number): MaskBox {
const sorted = [...region.keyframes].sort((a, b) => a.tMs - b.tMs);
let box = sorted[0]?.box ?? { x: 0, y: 0, w: 0, h: 0 };
for (const keyframe of sorted) if (keyframe.tMs <= sourceMs) box = keyframe.box;
return box;
}
/** Clamp a box into the frame with a minimum size. */
export function clampBox(box: MaskBox): MaskBox {
const w = Math.min(1, Math.max(0.02, box.w));
const h = Math.min(1, Math.max(0.02, box.h));
return {
x: Math.min(1 - w, Math.max(0, box.x)),
y: Math.min(1 - h, Math.max(0, box.y)),
w,
h,
};
}
/** Add a mask to a clip with one keyframe at `sourceMs`. Returns the plan and the new id. */
export function addRegion(edl: Edl, clipId: string, sourceMs: number): { edl: Edl; id: string } {
const id = nextId(edl, "mask");
const region: RegionRedaction = {
id,
clipId,
treatment: "blur",
keyframes: [{ tMs: Math.round(sourceMs), box: { x: 0.4, y: 0.35, w: 0.2, h: 0.3 } }],
};
return {
edl: {
...edl,
redactions: { ...edl.redactions, regions: [...edl.redactions.regions, region] },
},
id,
};
}
/** Set a region's box at `sourceMs`, replacing a keyframe at that exact time or adding one. */
export function setRegionKeyframe(edl: Edl, regionId: string, sourceMs: number, box: MaskBox): Edl {
const tMs = Math.round(sourceMs);
const clamped = clampBox(box);
return {
...edl,
redactions: {
...edl.redactions,
regions: edl.redactions.regions.map((region) => {
if (region.id !== regionId) return region;
const others = region.keyframes.filter((keyframe) => keyframe.tMs !== tMs);
const keyframes = [...others, { tMs, box: clamped }].sort((a, b) => a.tMs - b.tMs);
return { ...region, keyframes };
}),
},
};
}
export function removeRegion(edl: Edl, regionId: string): Edl {
return {
...edl,
redactions: {
...edl.redactions,
regions: edl.redactions.regions.filter((region) => region.id !== regionId),
},
};
}
export type DragId =
| `clip:${string}`
| `trim-start:${string}`
| `trim-end:${string}`
| `overlay:${string}`;
/** Apply an in-flight drag (pixels converted to ms) to the plan. */
export function applyDrag(
edl: Edl,
id: DragId,
deltaMs: number,
assetDurations: ReadonlyMap<string, number | null>,
): Edl {
const [kind, target] = id.split(/:(.*)/s, 2) as [string, string];
const assetDuration = (clipId: string) => {
const clip = edl.tracks.video.find((candidate) => candidate.id === clipId);
return clip ? (assetDurations.get(clip.assetId) ?? null) : null;
};
switch (kind) {
case "clip":
return moveClip(edl, target, deltaMs);
case "trim-start":
return trimClip(edl, target, "start", deltaMs, assetDuration(target));
case "trim-end":
return trimClip(edl, target, "end", deltaMs, assetDuration(target));
case "overlay":
return moveOverlay(edl, target, deltaMs);
default:
return edl;
}
}
/** `1:05.3` style timestamp. */
export function formatMs(ms: number): string {
const tenths = Math.max(0, Math.round(ms / 100));
const minutes = Math.floor(tenths / 600);
const seconds = (tenths % 600) / 10;
return `${minutes}:${seconds.toFixed(1).padStart(4, "0")}`;
}

View File

@ -0,0 +1,88 @@
import { describe, expect, it } from "vitest";
import {
agentPointerFeed,
moveDurationMs,
planPointerTimeline,
pointerFrameAt,
} from "./agent-pointer";
const payload = {
kind: "pointer",
screen: { width: 1280, height: 800 },
steps: [{ action: "click", x: 100, y: 100 }],
};
describe("agentPointerFeed", () => {
it("delivers valid live announcements only to the bot's viewers", () => {
const seen: unknown[] = [];
const other: unknown[] = [];
const stop = agentPointerFeed.subscribe("bot-a", (value) => seen.push(value));
const stopOther = agentPointerFeed.subscribe("bot-b", (value) => other.push(value));
try {
const now = Date.parse("2026-01-01T00:00:10Z");
expect(agentPointerFeed.publish("bot-a", payload, "2026-01-01T00:00:05Z", now)).toBe(true);
// Replayed history and malformed payloads never move the cursor.
expect(agentPointerFeed.publish("bot-a", payload, "2025-12-31T23:00:00Z", now)).toBe(false);
expect(agentPointerFeed.publish("bot-a", { ...payload, steps: [] }, undefined, now)).toBe(
false,
);
expect(agentPointerFeed.publish("bot-a", { kind: "status" }, undefined, now)).toBe(false);
expect(seen).toEqual([payload]);
expect(other).toEqual([]);
} finally {
stop();
stopOther();
}
expect(agentPointerFeed.publish("bot-a", payload)).toBe(true);
expect(seen).toHaveLength(1);
});
});
describe("pointer timeline", () => {
it("travels, acts in place for keyboard steps, then rests", () => {
const segments = planPointerTimeline(
{ x: 0, y: 0 },
[{ action: "click", x: 300, y: 400 }, { action: "type" }, { action: "move", x: 300, y: 400 }],
1_000,
);
expect(segments.map((segment) => segment.to)).toEqual([
{ x: 300, y: 400 },
{ x: 300, y: 400 },
{ x: 300, y: 400 },
]);
const [click, type, move] = segments;
expect(click!.arrive - click!.start).toBe(moveDurationMs(500));
expect(type!.arrive).toBe(type!.start);
expect(move!.end).toBe(move!.start);
const midway = pointerFrameAt(segments, (click!.start + click!.arrive) / 2)!;
expect(midway.action).toBeNull();
// The bow lifts the cursor off the straight line between the points.
const onLine = { x: 150, y: 200 };
expect(Math.hypot(midway.point.x - onLine.x, midway.point.y - onLine.y)).toBeGreaterThan(10);
const clicking = pointerFrameAt(segments, click!.arrive + 1)!;
expect(clicking.point).toEqual({ x: 300, y: 400 });
expect(clicking.action?.kind).toBe("click");
const typing = pointerFrameAt(segments, type!.arrive + 10)!;
expect(typing.action?.kind).toBe("type");
expect(pointerFrameAt(segments, move!.end + 1)).toEqual({
point: { x: 300, y: 400 },
action: null,
done: true,
});
});
it("jumps straight to the target when motion is reduced", () => {
const segments = planPointerTimeline({ x: 0, y: 0 }, [{ action: "move", x: 50, y: 60 }], 0);
expect(pointerFrameAt(segments, 1, true)?.point).toEqual({ x: 50, y: 60 });
});
it("bounds travel time so long and short moves both feel responsive", () => {
expect(moveDurationMs(0)).toBe(0);
expect(moveDurationMs(5)).toBe(180);
expect(moveDurationMs(10_000)).toBe(650);
});
});

View File

@ -0,0 +1,156 @@
import {
type ComputerPointerAction,
ComputerPointerPayload,
type ComputerPointerStep,
} from "@rakazo/contracts";
export type PointerPoint = { x: number; y: number };
export type PointerScreen = ComputerPointerPayload["screen"];
/** Older announcements describe actions that already finished; replaying them would mislead. */
export const POINTER_EVENT_MAX_AGE_MS = 30_000;
export const POINTER_IDLE_MS = 3_500;
type Listener = (payload: ComputerPointerPayload) => void;
/**
* Live pointer announcements per bot. The thread subscription publishes `thread.computer`
* pointer events here; screen viewers subscribe for the bot they show.
*/
class AgentPointerFeed {
private readonly listeners = new Map<string, Set<Listener>>();
publish(botId: string, payload: unknown, createdAt?: string, now = Date.now()): boolean {
const parsed = ComputerPointerPayload.safeParse(payload);
if (!parsed.success) return false;
const at = createdAt ? Date.parse(createdAt) : Number.NaN;
if (Number.isFinite(at) && now - at > POINTER_EVENT_MAX_AGE_MS) return false;
for (const listener of this.listeners.get(botId) ?? []) listener(parsed.data);
return true;
}
subscribe(botId: string, listener: Listener): () => void {
let set = this.listeners.get(botId);
if (!set) {
set = new Set();
this.listeners.set(botId, set);
}
set.add(listener);
return () => {
set.delete(listener);
if (set.size === 0) this.listeners.delete(botId);
};
}
}
export const agentPointerFeed = new AgentPointerFeed();
export type PointerSegment = {
from: PointerPoint;
to: PointerPoint;
/** Perpendicular bow in screen pixels so long moves read as a hand, not a teleport. */
bow: number;
start: number;
arrive: number;
end: number;
action: ComputerPointerAction;
};
const ACTION_DWELL_MS: Record<ComputerPointerAction, number> = {
move: 0,
click: 320,
double_click: 440,
right_click: 320,
down: 160,
up: 160,
type: 700,
key: 280,
scroll: 420,
};
export function moveDurationMs(distance: number): number {
return distance < 1 ? 0 : Math.round(Math.min(650, Math.max(180, 140 + distance * 0.45)));
}
/**
* Lays announced steps on a timeline that starts where the cursor currently is. Keyboard
* steps carry no coordinates and act in place.
*/
export function planPointerTimeline(
from: PointerPoint,
steps: readonly ComputerPointerStep[],
startAt: number,
): PointerSegment[] {
const segments: PointerSegment[] = [];
let position = from;
let time = startAt;
for (const step of steps) {
const to =
typeof step.x === "number" && typeof step.y === "number"
? { x: step.x, y: step.y }
: position;
const distance = Math.hypot(to.x - position.x, to.y - position.y);
const arrive = time + moveDurationMs(distance);
const end = arrive + ACTION_DWELL_MS[step.action];
segments.push({
from: position,
to,
bow: Math.min(48, distance * 0.12),
start: time,
arrive,
end,
action: step.action,
});
position = to;
time = end;
}
return segments;
}
export type PointerFrame = {
point: PointerPoint;
/** The action being performed at the target, with 0..1 progress; null while travelling. */
action: { kind: ComputerPointerAction; progress: number } | null;
done: boolean;
};
function easeInOutCubic(t: number): number {
return t < 0.5 ? 4 * t * t * t : 1 - (-2 * t + 2) ** 3 / 2;
}
export function pointerFrameAt(
segments: readonly PointerSegment[],
now: number,
reducedMotion = false,
): PointerFrame | null {
if (segments.length === 0) return null;
const last = segments[segments.length - 1]!;
if (now >= last.end) return { point: last.to, action: null, done: true };
const segment = segments.find((candidate) => now < candidate.end) ?? last;
if (now < segment.arrive) {
if (reducedMotion) return { point: segment.to, action: null, done: false };
const span = segment.arrive - segment.start;
const t = easeInOutCubic(span > 0 ? Math.max(0, (now - segment.start) / span) : 1);
const dx = segment.to.x - segment.from.x;
const dy = segment.to.y - segment.from.y;
const length = Math.hypot(dx, dy) || 1;
const lift = Math.sin(Math.PI * t) * segment.bow;
return {
point: {
x: segment.from.x + dx * t - (dy / length) * lift,
y: segment.from.y + dy * t + (dx / length) * lift,
},
action: null,
done: false,
};
}
const dwell = segment.end - segment.arrive;
return {
point: segment.to,
action:
segment.action === "move"
? null
: { kind: segment.action, progress: dwell > 0 ? (now - segment.arrive) / dwell : 1 },
done: false,
};
}

View File

@ -1,36 +0,0 @@
import { describe, expect, it } from "vitest";
import { browserPreviewActivity } from "./browser-preview-context";
describe("browserPreviewActivity", () => {
it("publishes cursors to subscribers and drops cleared ids", () => {
for (const id of browserPreviewActivity.snapshot().keys()) browserPreviewActivity.clear(id);
const seen: string[][] = [];
const unsubscribe = browserPreviewActivity.subscribe((activities) => {
seen.push([...activities.keys()]);
});
try {
browserPreviewActivity.set("bot-a", 50, 40, "Ada");
browserPreviewActivity.set("bot-a", 50, 40, "Ada");
browserPreviewActivity.set("bot-b", 10, 20, "Bob");
expect(browserPreviewActivity.snapshot().get("bot-a")).toEqual({
x: 50,
y: 40,
label: "Ada",
});
browserPreviewActivity.clear("bot-a");
browserPreviewActivity.clear("missing");
expect(seen).toEqual([[], ["bot-a"], ["bot-a", "bot-b"], ["bot-b"]]);
} finally {
unsubscribe();
for (const id of [...browserPreviewActivity.snapshot().keys()]) {
browserPreviewActivity.clear(id);
}
}
expect(browserPreviewActivity.snapshot().size).toBe(0);
});
});

View File

@ -1,47 +0,0 @@
export type BrowserPreviewCursor = {
x: number;
y: number;
label: string;
};
type Listener = (activities: ReadonlyMap<string, BrowserPreviewCursor>) => void;
/**
* In-memory cursors for the desktop browser-preview panel.
* The shell publishes working bots here; the panel subscribes.
*/
class BrowserPreviewActivity {
private readonly activities = new Map<string, BrowserPreviewCursor>();
private readonly listeners = new Set<Listener>();
subscribe(listener: Listener): () => void {
this.listeners.add(listener);
listener(this.snapshot());
return () => {
this.listeners.delete(listener);
};
}
snapshot(): ReadonlyMap<string, BrowserPreviewCursor> {
return new Map(this.activities);
}
set(id: string, x: number, y: number, label: string) {
const current = this.activities.get(id);
if (current && current.x === x && current.y === y && current.label === label) return;
this.activities.set(id, { x, y, label });
this.emit();
}
clear(id: string) {
if (!this.activities.delete(id)) return;
this.emit();
}
private emit() {
const snapshot = this.snapshot();
for (const listener of this.listeners) listener(snapshot);
}
}
export const browserPreviewActivity = new BrowserPreviewActivity();

View File

@ -0,0 +1,38 @@
import { describe, expect, it } from "vitest";
import { browserFrameSandbox, normalizeBrowserUrl } from "./browser-url";
describe("normalizeBrowserUrl", () => {
it("defaults local dev servers to http and the web to https", () => {
expect(normalizeBrowserUrl("localhost:5173")).toBe("http://localhost:5173/");
expect(normalizeBrowserUrl("127.0.0.1:3000/app?x=1")).toBe("http://127.0.0.1:3000/app?x=1");
expect(normalizeBrowserUrl("example.com/docs")).toBe("https://example.com/docs");
expect(normalizeBrowserUrl(" https://example.com ")).toBe("https://example.com/");
expect(normalizeBrowserUrl("http://example.com")).toBe("http://example.com/");
});
it("rejects input that is not an http(s) address", () => {
for (const input of [
"",
"hello world",
"hello",
"javascript:alert(1)",
"file:///etc/passwd",
"data:text/html,hi",
"ftp://example.com",
"https://user:fake-password@example.com",
]) {
expect(normalizeBrowserUrl(input), input).toBeNull();
}
});
});
describe("browserFrameSandbox", () => {
it("never grants same-origin access to the app's own origin", () => {
expect(browserFrameSandbox("https://app.test/x", "https://app.test")).not.toContain(
"allow-same-origin",
);
expect(browserFrameSandbox("http://localhost:5173/", "https://app.test")).toContain(
"allow-same-origin",
);
});
});

View File

@ -0,0 +1,39 @@
const LOCAL_HOST = /^(localhost|127(?:\.\d{1,3}){3}|\[::1\]|0\.0\.0\.0|[\w-]+\.local(?:host)?)$/i;
/**
* Turns what someone types in the address bar into an http(s) URL. Local dev servers default to
* http, everything else to https. Returns null for input that is not an address.
*/
export function normalizeBrowserUrl(input: string): string | null {
const value = input.trim();
if (!value || /\s/.test(value)) return null;
const hasScheme = /^[a-z][a-z\d+.-]*:\/\//i.test(value);
if (!hasScheme && /^[a-z][a-z\d+.-]*:(?!\d)/i.test(value)) return null;
let candidate = value;
if (!hasScheme) {
const host = value.split(/[/?#]/, 1)[0]!.replace(/:\d+$/, "");
if (!LOCAL_HOST.test(host) && !host.includes(".") && !/^\d+$/.test(host)) return null;
candidate = `${LOCAL_HOST.test(host) ? "http" : "https"}://${value}`;
}
try {
const url = new URL(candidate);
if (url.protocol !== "http:" && url.protocol !== "https:") return null;
if (url.username || url.password || !url.hostname) return null;
return url.toString();
} catch {
return null;
}
}
/**
* Sandbox for the web fallback. A page on the app's own origin must not get same-origin
* access, or it could reach into the app around the sandbox.
*/
export function browserFrameSandbox(url: string, appOrigin: string): string {
const base = "allow-scripts allow-forms allow-popups allow-modals allow-downloads";
try {
return new URL(url).origin === appOrigin ? base : `${base} allow-same-origin`;
} catch {
return base;
}
}

View File

@ -925,6 +925,24 @@ describe("thread event reduction", () => {
expect(next?.messages).toEqual(initial.messages);
});
it("advances past a bot pointer announcement without touching messages", () => {
const initial = snapshot([message("m1", [{ kind: "text", text: "hi" }], 3)], 3);
const pointer = event({
type: "thread.computer",
seq: 4,
runId: "run-1",
payload: {
kind: "pointer",
screen: { width: 1280, height: 800 },
steps: [{ action: "click", x: 10, y: 20 }],
},
});
expect(isThreadSnapshotEvent(pointer)).toBe(true);
const next = reduceThreadSnapshot(initial, pointer);
expect(next?.cursor).toBe(4);
expect(next?.messages).toEqual(initial.messages);
});
it("holds a tool call that lands mid-sentence until the sentence completes", () => {
const initial = snapshot([]);

View File

@ -249,6 +249,7 @@ export function isThreadSnapshotEvent(event: ProductEvent): boolean {
event.type === "thread.cloud_agent" ||
event.type === "agent.tool.called" ||
event.type === "agent.tool.completed" ||
event.type === "thread.computer" ||
event.type === "thread.message.created" ||
event.type === "thread.message.updated" ||
event.type === "thread.message.reaction" ||
@ -439,7 +440,7 @@ export function reduceThreadSnapshot(
};
return { ...prev, cursor: event.seq, messages: [...remaining, next] };
}
if (event.type === "agent.tool.completed") {
if (event.type === "agent.tool.completed" || event.type === "thread.computer") {
return { ...prev, cursor: event.seq };
}
if (event.type === "thread.subagent") {

View File

@ -110,6 +110,7 @@ import {
Copy,
Folder,
Gauge,
Globe,
GripVertical,
LayoutGrid,
Lock,
@ -159,6 +160,7 @@ import { AskCard } from "../components/AskCard";
import { ActiveBotGlyph, CollaborationMarker } from "../components/ai/CollaborationMarker";
import { DelegatedTaskCard, SentMessageCard } from "../components/ai/DelegatedTaskCard";
import { ProjectCreatedCard } from "../components/ai/ProjectCreatedCard";
import { InAppBrowser } from "../components/browser/InAppBrowser";
import { CloudAgentCard } from "../components/CloudAgentCard";
import { ComputerMaintenanceActions } from "../components/ComputerMaintenanceActions";
import {
@ -166,6 +168,7 @@ import {
computersAreUnavailable,
} from "../components/ComputersUnavailableHint";
import { ComputerUpdateProgress } from "../components/ComputerUpdateProgress";
import { AgentCursor } from "../components/computer/AgentCursor";
import { DataTrustPanel } from "../components/DataTrustPanel";
import { MainInbox } from "../components/MainInbox";
import { MessageHoverMetadata } from "../components/MessageHoverMetadata";
@ -179,6 +182,7 @@ import { VoiceAuroraRim } from "../components/voice/VoiceAurora";
import { useSidebarResize } from "../hooks/useSidebarResize";
import { readActivityMode, writeActivityMode } from "../lib/activity-mode";
import { needsYouChipLabel, rosterActivitySubtitle, rosterStatusDot } from "../lib/agent-activity";
import { agentPointerFeed } from "../lib/agent-pointer";
import type { ArtifactTarget } from "../lib/artifact-open";
import { authClient } from "../lib/auth";
import { takeInitialBootstrap } from "../lib/bootstrap";
@ -192,8 +196,6 @@ import {
requestBrowserNotificationPermission,
shouldNotifyBrowser,
} from "../lib/browser-notifications";
import type { BrowserPreviewCursor } from "../lib/browser-preview-context";
import { browserPreviewActivity } from "../lib/browser-preview-context";
import { loadComputerScreen } from "../lib/computer-screen";
import { desktopBridge } from "../lib/desktop";
import { scheduleFocusPrompt } from "../lib/focus-prompt";
@ -381,31 +383,6 @@ function readCollapsedSidebarSections(userId: string | null | undefined): Set<st
}
}
function copyBrowserPreviewActivities(
activities: ReadonlyMap<string, BrowserPreviewCursor>,
): Map<string, BrowserPreviewCursor> {
return new Map(activities);
}
function sameBrowserPreviewActivities(
current: ReadonlyMap<string, BrowserPreviewCursor>,
next: ReadonlyMap<string, BrowserPreviewCursor>,
): boolean {
if (current.size !== next.size) return false;
for (const [id, cursor] of next) {
const existing = current.get(id);
if (
!existing ||
existing.x !== cursor.x ||
existing.y !== cursor.y ||
existing.label !== cursor.label
) {
return false;
}
}
return true;
}
export type NavPillar = "heute" | "team" | "projekte";
function stripSpaceTitlePrefix(title: string | null, spaceName: string): string | null {
@ -480,16 +457,7 @@ export function ShellPage() {
setBotsSidebarCollapsed(readBotsSidebarCollapsed(userId));
}, [userId]);
// ── Browser Preview ──────────────────────────────────────────────
const bridge = desktopBridge();
const [isDesktop] = useState(Boolean(bridge?.browserPreview));
const [panel, setPanel] = useState<Panel>(null);
const [browserUrl, setBrowserUrl] = useState("http://localhost:5173");
const [devServerRunning, setDevServerRunning] = useState(false);
const [browserLoading, setBrowserLoading] = useState(false);
const [agentActivities, setAgentActivities] = useState<
Map<string, { x: number; y: number; label: string }>
>(new Map());
const [showCreateBotModal, setShowCreateBotModal] = useState(false);
@ -505,60 +473,6 @@ export function ShellPage() {
useSensor(TouchSensor, { activationConstraint: { distance: 8 } }),
);
// Browser Preview lifecycle
useEffect(() => {
const bp = bridge?.browserPreview;
if (!bp) return;
const unsubNav = bp.onNavigate((url) => setBrowserUrl(url));
const unsubLoading = bp.onLoadingProgress((p) => setBrowserLoading(p < 1));
bp.getState()
.then((s) => {
setBrowserUrl(s.url);
setDevServerRunning(s.devServerRunning);
})
.catch(() => {});
return () => {
unsubNav();
unsubLoading();
};
}, [bridge]);
const startDevServer = useCallback(async () => {
try {
await bridge?.browserPreview?.devServer.start();
setDevServerRunning(true);
} catch {
/* */
}
}, [bridge]);
const stopDevServer = useCallback(async () => {
try {
await bridge?.browserPreview?.devServer.stop();
setDevServerRunning(false);
} catch {
/* */
}
}, [bridge]);
const navigateBrowser = useCallback(
(url: string) => {
bridge?.browserPreview?.navigate(url).catch(() => {});
},
[bridge],
);
const reloadBrowser = useCallback(() => {
bridge?.browserPreview?.reload().catch(() => {});
}, [bridge]);
// Agent activity → browser preview cursors
useEffect(() => {
const unsub = browserPreviewActivity.subscribe((acts) => {
setAgentActivities((current) =>
sameBrowserPreviewActivities(current, acts) ? current : copyBrowserPreviewActivities(acts),
);
});
return unsub;
}, []);
const [query, setQuery] = useState("");
const [searchHits, setSearchHits] = useState<SearchHit[]>([]);
const [searchLoading, setSearchLoading] = useState(false);
@ -1542,6 +1456,8 @@ export function ShellPage() {
void refreshThread(active.id).catch(() => undefined);
} else if (isComputerStatusEvent(event)) {
void refreshComputerScreen(active.id).catch(() => undefined);
} else if (event.type === "thread.computer") {
agentPointerFeed.publish(event.botId, event.payload, event.createdAt);
}
},
});
@ -2237,20 +2153,6 @@ export function ShellPage() {
}));
}, [activeSnapshot, enrichStageMember, inGroup, resolveTranscriptBot, workingRuns]);
// Working bots → browser preview activity cursors
useEffect(() => {
if (workingBots.length === 0) return;
for (const bot of workingBots) {
if (!bot.botId) continue;
const label = bot.name ?? bot.botId.slice(0, 8);
browserPreviewActivity.set(bot.botId, 50, 40, label);
}
const activeBotIds = new Set(workingBots.map((b) => b.botId).filter(Boolean));
for (const id of browserPreviewActivity.snapshot().keys()) {
if (!activeBotIds.has(id)) browserPreviewActivity.clear(id);
}
}, [workingBots]);
const encounterMode = Boolean((active || activeGroup) && !showProjectTimeline && !callOpen);
const voiceEncounterActive = encounterMode || callOpen || voiceEntry;
const voiceCallBot = useMemo(() => {
@ -4492,6 +4394,16 @@ export function ShellPage() {
</button>
</div>
<div className="flex items-center gap-2">
<button
type="button"
title={t`Browser`}
aria-label={t`Browser`}
onClick={() => setPanel(panel === "browser" ? null : "browser")}
data-active={panel === "browser" ? "" : undefined}
className="app-no-drag grid h-[30px] w-[34px] place-items-center rounded-[9px] hover:bg-accent data-active:bg-accent"
>
<Globe strokeWidth={1.7} className="text-foreground/75 size-icon-lg" />
</button>
{!inGroup && active ? (
<button
type="button"
@ -4721,7 +4633,9 @@ export function ShellPage() {
onPointerDown={startResizingRight}
className="absolute -left-2 top-0 bottom-0 w-4 cursor-col-resize z-50 hidden md:block"
/>
{panel && (active || activeGroup || panel === "create" || panel === "create-group") ? (
{panel === "browser" ? (
<InAppBrowser onClose={() => setPanel(null)} />
) : panel && (active || activeGroup || panel === "create" || panel === "create-group") ? (
<div className="rk-scroll h-full w-full overflow-y-auto px-5 lg:px-7 2xl:px-9 py-[17px]">
{panel !== "routine" &&
panel !== "create" &&
@ -4761,17 +4675,6 @@ export function ShellPage() {
<Settings strokeWidth={1.7} className="size-icon-md" />
</Button>
) : null}
{!panel && active ? (
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Browser preview`}
onClick={() => setPanel("browser")}
className="text-muted-foreground"
>
<PanelRight strokeWidth={1.7} className="size-icon-md" />
</Button>
) : null}
<Button
variant="ghost"
size="icon-sm"
@ -4783,80 +4686,6 @@ export function ShellPage() {
</div>
</div>
) : null}
{panel === "browser" ? (
<div className="flex h-full flex-col">
<div className="flex items-center gap-1.5 border-b px-3 py-2">
<Button
variant="ghost"
size="icon-sm"
onClick={reloadBrowser}
aria-label={t`Reload`}
className="size-icon-sm text-muted-foreground"
>
<RefreshCw strokeWidth={1.7} className="size-4" />
</Button>
<input
type="url"
value={browserUrl}
onChange={(e) => setBrowserUrl(e.target.value)}
onBlur={() => navigateBrowser(browserUrl)}
onKeyDown={(e) => {
if (e.key === "Enter") navigateBrowser(browserUrl);
}}
placeholder="http://localhost:5173"
className="flex-1 rounded-md bg-muted/60 px-2.5 py-1 text-xs text-foreground placeholder:text-muted-foreground/60 outline-none focus:ring-1 focus:ring-ring/50"
/>
{isDesktop ? (
devServerRunning ? (
<span className="inline-flex items-center gap-1 rounded-full bg-muted px-2 py-0.5 text-ui-label font-medium text-foreground">
<span className="size-1.5 rounded-full bg-success" />
<Trans>Dev</Trans>
</span>
) : (
<Button
variant="ghost"
size="xs"
onClick={startDevServer}
className="text-ui-label text-muted-foreground hover:text-foreground"
>
<Trans>Start Dev</Trans>
</Button>
)
) : null}
{browserLoading && (
<div className="h-0.5 w-full rounded-full bg-muted/30">
<div className="h-full w-2/3 animate-pulse rounded-full bg-primary/60" />
</div>
)}
</div>
<div className="relative flex-1 overflow-hidden bg-background">
<div className="relative h-full w-full" data-testid="browser-preview">
{agentActivities.size > 0 && (
<div className="pointer-events-none absolute inset-0">
{Array.from(agentActivities.entries()).map(([agentId, activity]) => {
const cx = Math.min(100, Math.max(0, activity.x)) + "%";
const cy = Math.min(100, Math.max(0, activity.y)) + "%";
return (
<div
key={agentId}
className="absolute transition-[left,top] duration-300 ease-out"
style={{ left: cx, top: cy }}
>
<div className="relative">
<div className="size-3 rounded-full bg-primary/60 ring-2 ring-primary" />
<div className="absolute -right-1 -top-4 whitespace-nowrap rounded bg-overlay px-1.5 py-0.5 text-ui-label text-foreground shadow-sm">
{activity.label}
</div>
</div>
</div>
);
})}
</div>
)}
</div>
</div>
</div>
) : null}
{panel === "computer" && active ? (
<div>
<div
@ -4870,14 +4699,17 @@ export function ShellPage() {
) : computer?.kind === "desktop" ? (
<DesktopKindEmptyState className="grid h-full place-items-center px-6 text-center text-sm text-muted-foreground/80" />
) : computer?.state === "running" && embeddedScreenUrl && !computerScreenError ? (
<iframe
title={t`Bot screen preview`}
src={embeddedScreenUrl}
sandbox={screenIframeSandbox(embeddedScreenUrl)}
className="h-full w-full border-0 bg-black"
allow="clipboard-read; clipboard-write"
style={{ pointerEvents: "none" }}
/>
<>
<iframe
title={t`Bot screen preview`}
src={embeddedScreenUrl}
sandbox={screenIframeSandbox(embeddedScreenUrl)}
className="h-full w-full border-0 bg-black"
allow="clipboard-read; clipboard-write"
style={{ pointerEvents: "none" }}
/>
<AgentCursor botId={active.id} color={active.color} name={active.name} />
</>
) : (
<div className="grid h-full place-items-center px-6 text-center text-sm text-muted-foreground/80">
{computerScreenError ??
@ -5482,6 +5314,7 @@ export function ShellPage() {
{newSpaceOpen ? (
<NewSpaceDialog
variant={newSpaceDialogVariant}
canLinkFolder={bootstrapMe?.isDeploymentOwner === true}
onCancel={() => setNewSpaceOpen(false)}
onConfirm={async ({ name, workspacePath }) => {
const space = await rpc.spaces.create({
@ -5767,6 +5600,9 @@ export function ShellPage() {
pointerEvents: recordingSkill || !hasControl ? "none" : "auto",
}}
/>
{active && !hasControl ? (
<AgentCursor botId={active.id} color={active.color} name={active.name} />
) : null}
{active ? (
<TeachCaptureOverlay
botId={active.id}

View File

@ -0,0 +1,587 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
BrandKit,
BrandKitRecord,
CaptionStylePreset,
IntroVariant,
MotionTemplateInfo,
Watermark,
} from "@rakazo/contracts";
import { Button, Card, CardContent, Field, FieldLabel, Input, Skeleton } from "@rakazo/ui-web";
import { Check, Plus, Save, Sparkles, Trash2 } from "lucide-react";
import { useCallback, useEffect, useId, useState } from "react";
import { LoadingState, SuccessPop } from "../../components/ai/primitives";
import { ErrorLine, errorMessage, ViewHeader } from "../../components/media/editor/chrome";
import {
NumberField,
SelectField,
TextField,
ToggleField,
} from "../../components/media/editor/fields";
import { mediaArtifactUrl, useMediaObjectUrl } from "../../components/media/editor/media-files";
import { formatMs, freeId } from "../../components/media/editor/timeline";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
/** Starting point for a new kit. Brand colors are render data, not UI colors. */
function emptyKit(name: string): BrandKit {
return {
name,
colors: {
primary: "#111111",
secondary: "#555555",
accent: "#999999",
text: "#ffffff",
background: "#000000",
},
fonts: [],
logoArtifactIds: [],
captionPresets: [],
musicBeds: [],
};
}
/** Brand kits: a list without an id, the kit editor and intro generator with one. */
export function BrandKitView({ navigate, brandKitId }: MediaViewProps & { brandKitId?: string }) {
const { t } = useLingui();
const [records, setRecords] = useState<BrandKitRecord[] | null>(null);
const [templates, setTemplates] = useState<MotionTemplateInfo[]>([]);
const [error, setError] = useState<string | null>(null);
const load = useCallback(async () => {
try {
setRecords(await rpc.media.brandKits.list({}));
} catch (cause) {
setError(errorMessage(cause));
}
}, []);
useEffect(() => {
void load();
void rpc.media.brandKits
.templates({})
.then(setTemplates)
.catch(() => undefined);
}, [load]);
if (!records) {
return (
<div className="flex flex-col gap-4 p-4">
<ErrorLine error={error} />
{error ? null : <Skeleton className="h-48 w-full" />}
</div>
);
}
const record = brandKitId ? records.find((candidate) => candidate.id === brandKitId) : undefined;
if (brandKitId && record) {
return (
<KitEditor
key={record.id}
record={record}
templates={templates}
onBack={() => navigate({ name: "brandKit" })}
onSaved={(saved) => setRecords((current) => upsertRecord(current ?? [], saved))}
/>
);
}
if (brandKitId === "new" || records.length === 0) {
return (
<KitEditor
record={null}
templates={templates}
onBack={() => navigate({ name: "library" })}
onSaved={(saved) => {
setRecords((current) => upsertRecord(current ?? [], saved));
navigate({ name: "brandKit", brandKitId: saved.id });
}}
/>
);
}
return (
<div className="flex flex-col gap-4 p-4" data-testid="media-brand-kits">
<ViewHeader
title={t`Brand kits`}
backLabel={t`Back to library`}
onBack={() => navigate({ name: "library" })}
>
<Button size="sm" onClick={() => navigate({ name: "brandKit", brandKitId: "new" })}>
<Plus />
<Trans>New</Trans>
</Button>
</ViewHeader>
<div className="grid gap-3 [grid-template-columns:repeat(auto-fill,minmax(14rem,1fr))]">
{records.map((item) => (
<button
key={item.id}
type="button"
className="text-left"
onClick={() => navigate({ name: "brandKit", brandKitId: item.id })}
>
<Card className="transition-colors hover:bg-muted/50">
<CardContent className="flex items-center gap-3">
<Swatches colors={item.kit.colors} />
<span className="min-w-0 flex-1 truncate text-ui-body font-medium">
{item.kit.name}
</span>
</CardContent>
</Card>
</button>
))}
</div>
</div>
);
}
function upsertRecord(records: BrandKitRecord[], saved: BrandKitRecord): BrandKitRecord[] {
return records.some((item) => item.id === saved.id)
? records.map((item) => (item.id === saved.id ? saved : item))
: [...records, saved];
}
function Swatches({ colors }: { colors: BrandKit["colors"] }) {
return (
<span aria-hidden className="flex -space-x-1">
{[colors.primary, colors.secondary, colors.accent].map((color, index) => (
<span
key={index}
className="size-5 rounded-full border border-border"
style={{ backgroundColor: color }}
/>
))}
</span>
);
}
function KitEditor({
record,
templates,
onBack,
onSaved,
}: {
record: BrandKitRecord | null;
templates: MotionTemplateInfo[];
onBack: () => void;
onSaved: (record: BrandKitRecord) => void;
}) {
const { t } = useLingui();
const [kit, setKit] = useState<BrandKit>(() => record?.kit ?? emptyKit(t`Brand kit`));
const [dirty, setDirty] = useState(record === null);
const [saving, setSaving] = useState(false);
const [warnings, setWarnings] = useState<string[]>([]);
const [error, setError] = useState<string | null>(null);
const patch = (next: Partial<BrandKit>) => {
setKit((current) => ({ ...current, ...next }));
setDirty(true);
};
async function save(next: BrandKit = kit) {
setSaving(true);
setError(null);
try {
const result = await rpc.media.brandKits.upsert({
...(record ? { id: record.id } : {}),
kit: next,
});
setKit(result.record.kit);
setWarnings(result.warnings);
setDirty(false);
onSaved(result.record);
} catch (cause) {
setError(errorMessage(cause));
} finally {
setSaving(false);
}
}
const colorLabels: Record<keyof BrandKit["colors"], string> = {
primary: t`Primary`,
secondary: t`Secondary`,
accent: t`Accent`,
text: t`Text`,
background: t`Background`,
};
return (
<div className="flex flex-col gap-6 p-4" data-testid="media-brand-kit">
<ViewHeader title={kit.name} backLabel={t`Back`} onBack={onBack}>
<Button size="sm" disabled={!dirty || saving} onClick={() => void save()}>
{saving ? (
<LoadingState label={t`Saving`} />
) : (
<>
<Save />
<Trans>Save</Trans>
</>
)}
</Button>
</ViewHeader>
<ErrorLine error={error} />
{warnings.map((warning) => (
<p key={warning} className="text-ui-body-sm text-warning">
{warning}
</p>
))}
<section className="grid max-w-2xl grid-cols-1 gap-3 sm:grid-cols-2">
<TextField
label={t`Name`}
value={kit.name}
maxLength={60}
onChange={(name) => patch({ name })}
/>
<TextField
label={t`Tone`}
value={kit.tone ?? ""}
maxLength={200}
onChange={(tone) => patch({ tone: tone || undefined })}
/>
<TextField
label={t`Audience`}
value={kit.audience ?? ""}
maxLength={200}
onChange={(audience) => patch({ audience: audience || undefined })}
/>
</section>
<section className="flex flex-wrap gap-4">
{(Object.keys(colorLabels) as Array<keyof BrandKit["colors"]>).map((key) => (
<ColorField
key={key}
label={colorLabels[key]}
value={kit.colors[key]}
onChange={(value) => patch({ colors: { ...kit.colors, [key]: value } })}
/>
))}
</section>
<WatermarkFields watermark={kit.watermark} onChange={(watermark) => patch({ watermark })} />
<CaptionPresets
presets={kit.captionPresets}
colors={kit.colors}
onChange={(captionPresets) => patch({ captionPresets })}
/>
{kit.fonts.length + kit.musicBeds.length > 0 ? (
<ul className="flex flex-col gap-1 text-ui-body-sm text-muted-foreground">
{kit.fonts.map((font) => (
<li key={font.artifactId}>
{font.family}
{font.license ? ` · ${font.license}` : ""}
</li>
))}
{kit.musicBeds.map((bed) => (
<li key={bed.artifactId}>
{bed.title} · {bed.license}
</li>
))}
</ul>
) : null}
{record ? (
<IntroGenerator
kitId={record.id}
kit={kit}
templates={templates}
disabled={dirty || saving}
onUse={(variant) => void save({ ...kit, intro: variant.template })}
/>
) : null}
</div>
);
}
function ColorField({
label,
value,
onChange,
}: {
label: string;
value: string;
onChange: (value: string) => void;
}) {
const id = useId();
return (
<Field className="w-24 gap-1">
<FieldLabel htmlFor={id}>{label}</FieldLabel>
<Input
id={id}
type="color"
className="h-control-md p-1"
value={value}
onChange={(event) => onChange(event.target.value)}
/>
</Field>
);
}
function WatermarkFields({
watermark,
onChange,
}: {
watermark: Watermark | undefined;
onChange: (watermark: Watermark | undefined) => void;
}) {
const { t } = useLingui();
return (
<section className="flex max-w-2xl flex-col gap-3">
<ToggleField
label={t`Watermark`}
checked={watermark !== undefined}
onChange={(on) =>
onChange(on ? { position: "bottom_right", opacity: 0.6, scale: 0.12 } : undefined)
}
/>
{watermark ? (
<div className="grid grid-cols-1 gap-3 sm:grid-cols-3">
<SelectField
label={t`Position`}
value={watermark.position}
options={[
{ value: "top_left", label: t`Top left` },
{ value: "top_right", label: t`Top right` },
{ value: "bottom_left", label: t`Bottom left` },
{ value: "bottom_right", label: t`Bottom right` },
{ value: "center", label: t`Center` },
]}
onChange={(position) => onChange({ ...watermark, position })}
/>
<NumberField
label={t`Opacity`}
value={watermark.opacity}
min={0}
max={1}
step={0.05}
onChange={(opacity) => onChange({ ...watermark, opacity })}
/>
<NumberField
label={t`Size`}
value={watermark.scale}
min={0.02}
max={0.5}
step={0.01}
onChange={(scale) => onChange({ ...watermark, scale })}
/>
</div>
) : null}
</section>
);
}
function CaptionPresets({
presets,
colors,
onChange,
}: {
presets: CaptionStylePreset[];
colors: BrandKit["colors"];
onChange: (presets: CaptionStylePreset[]) => void;
}) {
const { t } = useLingui();
const update = (id: string, next: Partial<CaptionStylePreset>) =>
onChange(presets.map((preset) => (preset.id === id ? { ...preset, ...next } : preset)));
return (
<section className="flex max-w-2xl flex-col gap-3" data-testid="media-caption-presets">
{presets.map((preset) => (
<div key={preset.id} className="grid grid-cols-2 items-end gap-3 sm:grid-cols-5">
<TextField
label={t`Caption style`}
value={preset.name}
maxLength={40}
onChange={(name) => update(preset.id, { name })}
/>
<SelectField
label={t`Look`}
value={preset.template}
options={[
{ value: "clean", label: t`Clean` },
{ value: "karaoke", label: t`Karaoke` },
{ value: "bold_social", label: t`Bold` },
]}
onChange={(template) => update(preset.id, { template })}
/>
<TextField
label={t`Font`}
value={preset.fontFamily}
maxLength={80}
onChange={(fontFamily) => update(preset.id, { fontFamily })}
/>
<ColorField
label={t`Color`}
value={preset.textColor}
onChange={(textColor) => update(preset.id, { textColor })}
/>
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Remove ${preset.name}`}
onClick={() => onChange(presets.filter((item) => item.id !== preset.id))}
>
<Trash2 />
</Button>
</div>
))}
<Button
variant="outline"
size="sm"
className="self-start"
disabled={presets.length >= 12}
onClick={() =>
onChange([
...presets,
{
id: freeId(
presets.map((preset) => preset.id),
"caption",
),
name: t`Captions`,
template: "clean",
fontFamily: "Inter",
sizePct: 5,
textColor: colors.text,
highlightColor: colors.accent,
uppercase: false,
},
])
}
>
<Plus />
<Trans>Caption style</Trans>
</Button>
</section>
);
}
function IntroGenerator({
kitId,
kit,
templates,
disabled,
onUse,
}: {
kitId: string;
kit: BrandKit;
templates: MotionTemplateInfo[];
disabled: boolean;
onUse: (variant: IntroVariant) => void;
}) {
const { t } = useLingui();
const [variants, setVariants] = useState<IntroVariant[]>([]);
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const templateName = (id: string) => templates.find((item) => item.id === id)?.name ?? id;
async function suggest() {
setBusy(true);
setError(null);
try {
setVariants(await rpc.media.brandKits.introVariants({ id: kitId, count: 3 }));
} catch (cause) {
setError(errorMessage(cause));
} finally {
setBusy(false);
}
}
const inUse = (variant: IntroVariant) =>
kit.intro?.templateId === variant.template.templateId &&
kit.intro.durationMs === variant.template.durationMs &&
JSON.stringify(kit.intro.params) === JSON.stringify(variant.template.params);
return (
<section className="flex flex-col gap-3" data-testid="media-intro-generator">
<div className="flex items-center gap-3">
<h3 className="text-ui-body font-semibold">
<Trans>Intro</Trans>
</h3>
{kit.intro ? (
<span className="text-ui-body-sm text-muted-foreground">
{templateName(kit.intro.templateId)} · {formatMs(kit.intro.durationMs)}
</span>
) : null}
<Button variant="outline" size="sm" disabled={busy || disabled} onClick={suggest}>
{busy ? (
<LoadingState label={t`Rendering previews`} />
) : (
<>
<Sparkles />
{variants.length > 0 ? <Trans>Suggest again</Trans> : <Trans>Suggest</Trans>}
</>
)}
</Button>
</div>
<ErrorLine error={error} />
{variants.length > 0 ? (
<div className="grid gap-3 [grid-template-columns:repeat(auto-fill,minmax(14rem,1fr))]">
{variants.map((variant) => (
<IntroCard
key={variant.id}
variant={variant}
name={templateName(variant.template.templateId)}
selected={inUse(variant)}
disabled={disabled}
onUse={() => onUse(variant)}
/>
))}
</div>
) : null}
</section>
);
}
function IntroCard({
variant,
name,
selected,
disabled,
onUse,
}: {
variant: IntroVariant;
name: string;
selected: boolean;
disabled: boolean;
onUse: () => void;
}) {
const { t } = useLingui();
const preview = useMediaObjectUrl(
variant.previewArtifactId ? mediaArtifactUrl(variant.previewArtifactId) : null,
);
return (
<Card data-testid="media-intro-variant">
<CardContent className="flex flex-col gap-2">
{preview ? (
<video
aria-label={t`${name} preview`}
className="aspect-video w-full rounded-md bg-muted object-contain"
src={preview}
autoPlay
loop
muted
playsInline
/>
) : (
<div className="aspect-video w-full rounded-md bg-muted" />
)}
<div className="flex items-baseline gap-2">
<span className="min-w-0 flex-1 truncate text-ui-body font-medium">{name}</span>
<span className="font-mono text-ui-body-sm tabular-nums text-muted-foreground">
{formatMs(variant.template.durationMs)}
</span>
</div>
<p className="text-ui-body-sm text-muted-foreground">{variant.rationale}</p>
{selected ? (
<SuccessPop label={t`In use`} />
) : (
<Button size="sm" variant="outline" disabled={disabled} onClick={onUse}>
<Check />
<Trans>Use</Trans>
</Button>
)}
</CardContent>
</Card>
);
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-library slice replaces this view. */
export function CollectionView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,450 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
CaptionStylePreset,
Edl,
EdlCaptions,
EdlError,
EdlOverlay,
MediaEditPlan,
} from "@rakazo/contracts";
import { Button, Skeleton } from "@rakazo/ui-web";
import { Plus, Save, Trash2 } from "lucide-react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import { ErrorLine, errorMessage, ViewHeader } from "../../components/media/editor/chrome";
import {
NumberField,
SecondsField,
SelectField,
TextField,
ToggleField,
} from "../../components/media/editor/fields";
import type { PlayerHandle } from "../../components/media/editor/Player";
import { SourcePlayer } from "../../components/media/editor/Player";
import type { TimelineSelection } from "../../components/media/editor/Timeline";
import { Timeline } from "../../components/media/editor/Timeline";
import {
errorTarget,
formatMs,
mainProgramMs,
nextId,
updateOverlay,
} from "../../components/media/editor/timeline";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
interface Loaded {
plan: MediaEditPlan;
assetDurations: Map<string, number | null>;
captionPresets: CaptionStylePreset[];
}
/** Timeline editor for one plan version. Saving submits a new version. */
export function EditorView({
navigate,
projectId,
planId,
}: MediaViewProps & { projectId: string; planId: string }) {
const { t } = useLingui();
const [loaded, setLoaded] = useState<Loaded | null>(null);
const [edl, setEdl] = useState<Edl | null>(null);
const [dirty, setDirty] = useState(false);
const [selection, setSelection] = useState<TimelineSelection>(null);
const [errors, setErrors] = useState<EdlError[]>([]);
const [playheadMs, setPlayheadMs] = useState(0);
const [saving, setSaving] = useState(false);
const [loadError, setLoadError] = useState<string | null>(null);
const player = useRef<PlayerHandle>(null);
useEffect(() => {
let live = true;
setLoaded(null);
setEdl(null);
setDirty(false);
setErrors([]);
setSelection(null);
void (async () => {
try {
const { project, plans } = await rpc.media.projects.get({ projectId });
const plan = plans.find((candidate) => candidate.id === planId);
if (!plan) throw new Error(t`This plan no longer exists.`);
const assetIds = [...new Set(plan.edl.tracks.video.map((clip) => clip.assetId))];
const [assets, kits] = await Promise.all([
Promise.all(
assetIds.map((assetId) =>
rpc.media.assets
.get({ assetId })
.then(({ asset }) => [assetId, asset.durationMs] as const)
.catch(() => [assetId, null] as const),
),
),
project.brandKitId ? rpc.media.brandKits.list({}) : Promise.resolve([]),
]);
if (!live) return;
const kit = kits.find((record) => record.id === project.brandKitId);
setLoaded({
plan,
assetDurations: new Map(assets),
captionPresets: kit?.kit.captionPresets ?? [],
});
setEdl(plan.edl);
} catch (cause) {
if (live) setLoadError(errorMessage(cause));
}
})();
return () => {
live = false;
};
}, [projectId, planId]);
const change = useCallback((next: Edl) => {
setEdl(next);
setDirty(true);
}, []);
const invalid = useMemo(() => {
const ids = new Set<string>();
if (!edl) return ids;
for (const error of errors) {
const target = errorTarget(edl, error);
if (target) ids.add(target.id);
}
return ids;
}, [edl, errors]);
if (!loaded || !edl) {
return (
<div className="flex flex-col gap-4 p-4">
<ErrorLine error={loadError} />
{loadError ? null : <Skeleton className="h-64 w-full" />}
</div>
);
}
async function save() {
if (!edl) return;
setSaving(true);
try {
const result = await rpc.media.plans.submit({ projectId, edl, rationale: null });
if (result.ok) {
setErrors([]);
setDirty(false);
navigate({ name: "editor", projectId, planId: result.plan.id });
} else {
setErrors(result.errors);
}
} catch (cause) {
setErrors([{ code: "schema", path: "", message: errorMessage(cause) }]);
} finally {
setSaving(false);
}
}
function addOverlay() {
if (!edl) return;
const id = nextId(edl, "text");
const atMs = Math.max(0, Math.min(playheadMs, mainProgramMs(edl)));
const overlay: EdlOverlay = { id, kind: "text", atMs, durationMs: 2000, text: "" };
change({ ...edl, overlays: [...edl.overlays, overlay] });
setSelection({ kind: "overlay", id });
}
return (
<div className="flex flex-col gap-4 p-4" data-testid="media-editor">
<ViewHeader
title={`${loaded.plan.profileId} · v${loaded.plan.version}`}
backLabel={t`Back to project`}
onBack={() => navigate({ name: "project", projectId })}
>
<Button size="sm" disabled={!dirty || saving} onClick={save}>
{saving ? (
<LoadingState label={t`Saving`} />
) : (
<>
<Save />
<Trans>Save</Trans>
</>
)}
</Button>
</ViewHeader>
<div className="mx-auto w-full max-w-2xl">
<SourcePlayer
edl={edl}
handleRef={player}
label={t`Plan preview`}
onFrame={(frame) => {
if (frame) setPlayheadMs(frame.outputMs);
}}
/>
</div>
<div className="flex items-center gap-2">
<span className="flex-1 font-mono text-ui-body-sm tabular-nums text-muted-foreground">
{formatMs(playheadMs)}
</span>
<Button variant="outline" size="sm" onClick={addOverlay}>
<Plus />
<Trans>Text</Trans>
</Button>
</div>
<Timeline
edl={edl}
assetDurations={loaded.assetDurations}
selection={selection}
invalid={invalid}
playheadMs={playheadMs}
onSelect={setSelection}
onSeek={(ms) => {
setPlayheadMs(ms);
player.current?.seek(ms);
}}
onChange={change}
/>
{errors.length > 0 ? (
<ul role="alert" className="flex flex-col gap-1" data-testid="media-edl-errors">
{errors.map((error) => {
const target = errorTarget(edl, error);
return (
<li key={`${error.code}-${error.path}-${error.message}`}>
<button
type="button"
disabled={!target}
onClick={() => target && setSelection(target)}
className="text-left text-ui-body-sm text-destructive enabled:hover:underline"
>
{error.message}
</button>
</li>
);
})}
</ul>
) : null}
<Inspector
edl={edl}
selection={selection}
assetDurations={loaded.assetDurations}
captionPresets={loaded.captionPresets}
onChange={change}
onClear={() => setSelection(null)}
/>
</div>
);
}
function Inspector({
edl,
selection,
assetDurations,
captionPresets,
onChange,
onClear,
}: {
edl: Edl;
selection: TimelineSelection;
assetDurations: ReadonlyMap<string, number | null>;
captionPresets: CaptionStylePreset[];
onChange: (edl: Edl) => void;
onClear: () => void;
}) {
const { t } = useLingui();
if (!selection) return null;
if (selection.kind === "clip") {
const clip = edl.tracks.video.find((candidate) => candidate.id === selection.id);
if (!clip) return null;
const max = assetDurations.get(clip.assetId) ?? Number.POSITIVE_INFINITY;
const patch = (next: Partial<typeof clip>) =>
onChange({
...edl,
tracks: {
...edl.tracks,
video: edl.tracks.video.map((candidate) =>
candidate.id === clip.id ? { ...candidate, ...next } : candidate,
),
},
});
return (
<section className="grid grid-cols-2 gap-3 sm:grid-cols-4" data-testid="media-inspector">
<SecondsField label={t`Start`} ms={clip.atMs} onChange={(atMs) => patch({ atMs })} />
<SecondsField
label={t`In`}
ms={clip.inMs}
onChange={(inMs) => patch({ inMs: Math.min(inMs, clip.outMs - 100) })}
/>
<SecondsField
label={t`Out`}
ms={clip.outMs}
onChange={(outMs) => patch({ outMs: Math.min(max, Math.max(outMs, clip.inMs + 100)) })}
/>
<NumberField
label={t`Speed`}
value={clip.speed}
min={0.25}
max={4}
step={0.25}
onChange={(speed) => patch({ speed })}
/>
<SelectField
label={t`Framing`}
value={Array.isArray(clip.reframe) ? "keyframes" : clip.reframe}
options={[
{ value: "auto", label: t`Follow subject` },
{ value: "none", label: t`Center` },
...(Array.isArray(clip.reframe)
? [{ value: "keyframes" as const, label: t`Keyframes` }]
: []),
]}
onChange={(value) => {
if (value !== "keyframes") patch({ reframe: value });
}}
/>
<ToggleField label={t`Mute`} checked={clip.muted} onChange={(muted) => patch({ muted })} />
{edl.tracks.video.length > 1 ? (
<Button
variant="ghost"
size="sm"
className="self-end"
onClick={() => {
onChange({
...edl,
tracks: {
...edl.tracks,
video: edl.tracks.video.filter((candidate) => candidate.id !== clip.id),
},
transitions: edl.transitions.filter(
(transition) =>
transition.fromClipId !== clip.id && transition.toClipId !== clip.id,
),
redactions: {
...edl.redactions,
regions: edl.redactions.regions.filter((region) => region.clipId !== clip.id),
},
});
onClear();
}}
>
<Trash2 />
<Trans>Remove</Trans>
</Button>
) : null}
</section>
);
}
if (selection.kind === "overlay") {
const overlay = edl.overlays.find((candidate) => candidate.id === selection.id);
if (!overlay) return null;
const patch = (next: Partial<EdlOverlay>) => onChange(updateOverlay(edl, overlay.id, next));
return (
<section className="grid grid-cols-2 gap-3 sm:grid-cols-4" data-testid="media-inspector">
<SelectField
label={t`Kind`}
value={overlay.kind}
options={[
{ value: "text", label: t`Text` },
{ value: "lower_third", label: t`Lower third` },
{ value: "logo", label: t`Logo` },
{ value: "watermark", label: t`Watermark` },
{ value: "cta_end_card", label: t`End card` },
{ value: "progress_bar", label: t`Progress bar` },
...(overlay.kind === "motion" ? [{ value: "motion" as const, label: t`Motion` }] : []),
]}
onChange={(kind) => patch({ kind })}
/>
<TextField
label={t`Text`}
value={overlay.text ?? ""}
maxLength={200}
onChange={(text) => patch({ text })}
/>
<SecondsField label={t`Start`} ms={overlay.atMs} onChange={(atMs) => patch({ atMs })} />
<SecondsField
label={t`Duration`}
ms={overlay.durationMs}
min={100}
onChange={(durationMs) => patch({ durationMs })}
/>
<Button
variant="ghost"
size="sm"
className="self-end"
onClick={() => {
onChange({
...edl,
overlays: edl.overlays.filter((candidate) => candidate.id !== overlay.id),
});
onClear();
}}
>
<Trash2 />
<Trans>Remove</Trans>
</Button>
</section>
);
}
const captions: EdlCaptions = edl.captions ?? {
mode: "none",
maxCharsPerLine: 32,
wordHighlight: false,
position: "lower",
};
const patch = (next: Partial<EdlCaptions>) =>
onChange({ ...edl, captions: { ...captions, ...next } });
return (
<section className="grid grid-cols-2 gap-3 sm:grid-cols-4" data-testid="media-inspector">
<SelectField
label={t`Captions`}
value={captions.mode}
options={[
{ value: "burned", label: t`In video` },
{ value: "sidecar", label: t`Subtitle file` },
{ value: "both", label: t`Both` },
{ value: "none", label: t`Off` },
]}
onChange={(mode) => patch({ mode })}
/>
{captions.mode === "none" ? null : (
<>
<SelectField
label={t`Position`}
value={captions.position}
options={[
{ value: "upper", label: t`Top` },
{ value: "middle", label: t`Middle` },
{ value: "lower", label: t`Bottom` },
]}
onChange={(position) => patch({ position })}
/>
<NumberField
label={t`Characters per line`}
value={captions.maxCharsPerLine}
min={10}
max={60}
onChange={(maxCharsPerLine) => patch({ maxCharsPerLine })}
/>
{captionPresets.length > 0 ? (
<SelectField
label={t`Style`}
value={captions.presetId ?? ""}
options={[
{ value: "", label: t`Default` },
...captionPresets.map((preset) => ({ value: preset.id, label: preset.name })),
]}
onChange={(presetId) => patch({ presetId: presetId || undefined })}
/>
) : null}
<ToggleField
label={t`Highlight words`}
checked={captions.wordHighlight}
onChange={(wordHighlight) => patch({ wordHighlight })}
/>
</>
)}
</section>
);
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-library slice replaces this view. */
export function LibraryView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,34 @@
import { useState } from "react";
import { BrandKitView } from "./BrandKitView";
import { CollectionView } from "./CollectionView";
import { EditorView } from "./EditorView";
import { LibraryView } from "./LibraryView";
import type { MediaView } from "./nav";
import { PeopleView } from "./PeopleView";
import { ProjectView } from "./ProjectView";
import { ResultsView } from "./ResultsView";
import { ReviewView } from "./ReviewView";
/** Media Studio surface. Chrome (header, close, tabs) belongs to the W3-web-library slice. */
export function MediaStudio({ onClose }: { onClose: () => void }) {
const [view, setView] = useState<MediaView>({ name: "library" });
const props = { navigate: setView, onClose };
switch (view.name) {
case "library":
return <LibraryView {...props} />;
case "collection":
return <CollectionView {...props} collectionId={view.collectionId} />;
case "people":
return <PeopleView {...props} />;
case "results":
return <ResultsView {...props} />;
case "project":
return <ProjectView {...props} projectId={view.projectId} />;
case "editor":
return <EditorView {...props} projectId={view.projectId} planId={view.planId} />;
case "review":
return <ReviewView {...props} projectId={view.projectId} renderId={view.renderId} />;
case "brandKit":
return <BrandKitView {...props} brandKitId={view.brandKitId} />;
}
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-library slice replaces this view. */
export function PeopleView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,292 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
MediaEditPlan,
MediaProject,
MediaRender,
PlatformProfileSummary,
} from "@rakazo/contracts";
import { Button, Card, CardContent, CardHeader, CardTitle, Skeleton } from "@rakazo/ui-web";
import { Download, Palette, Pencil, Play, ShieldCheck, Sparkles, X } from "lucide-react";
import { useCallback, useEffect, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import {
ErrorLine,
errorMessage,
GateBadge,
ViewHeader,
} from "../../components/media/editor/chrome";
import {
downloadMediaFile,
renderPosterUrl,
renderVideoUrl,
useMediaObjectUrl,
usePolling,
} from "../../components/media/editor/media-files";
import { RenderVideo } from "../../components/media/editor/Player";
import { PublishPanel } from "../../components/media/editor/PublishPanel";
import type { Variant } from "../../components/media/editor/renders";
import {
anyRenderActive,
gateStatus,
isRenderActive,
projectVariants,
RENDER_POLL_MS,
} from "../../components/media/editor/renders";
import { edlDurationMs, formatMs } from "../../components/media/editor/timeline";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
interface ProjectData {
project: MediaProject;
plans: MediaEditPlan[];
renders: MediaRender[];
}
/** A project's variants, one card per platform profile. */
export function ProjectView({ navigate, projectId }: MediaViewProps & { projectId: string }) {
const { t } = useLingui();
const [data, setData] = useState<ProjectData | null>(null);
const [profiles, setProfiles] = useState<Map<string, PlatformProfileSummary>>(new Map());
const [planning, setPlanning] = useState(false);
const [error, setError] = useState<string | null>(null);
const load = useCallback(async () => {
try {
setData(await rpc.media.projects.get({ projectId }));
} catch (cause) {
setError(errorMessage(cause));
}
}, [projectId]);
useEffect(() => {
void load();
void rpc.media.profiles
.list({})
.then((list) => setProfiles(new Map(list.map((profile) => [profile.id, profile]))))
.catch(() => undefined);
}, [load]);
usePolling(load, data ? anyRenderActive(data.renders) : false, RENDER_POLL_MS);
if (!data) {
return (
<div className="flex flex-col gap-4 p-4">
<ErrorLine error={error} />
{error ? null : <Skeleton className="h-64 w-full" />}
</div>
);
}
const variants = projectVariants(data.project.profileIds, data.plans, data.renders);
const unplanned = variants.some((variant) => !variant.plan);
async function autoPlan() {
setPlanning(true);
setError(null);
try {
await rpc.media.projects.autoPlan({ projectId });
await load();
} catch (cause) {
setError(errorMessage(cause));
} finally {
setPlanning(false);
}
}
return (
<div className="flex flex-col gap-4 p-4" data-testid="media-project">
<ViewHeader
title={data.project.title}
backLabel={t`Back to library`}
onBack={() => navigate({ name: "library" })}
>
{data.project.brandKitId ? (
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Brand kit`}
onClick={() =>
navigate({ name: "brandKit", brandKitId: data.project.brandKitId ?? undefined })
}
>
<Palette />
</Button>
) : null}
{unplanned ? (
<Button size="sm" disabled={planning} onClick={autoPlan}>
{planning ? (
<LoadingState label={t`Planning`} />
) : (
<>
<Sparkles />
<Trans>Plan</Trans>
</>
)}
</Button>
) : null}
</ViewHeader>
<ErrorLine error={error} />
<div className="grid gap-4 [grid-template-columns:repeat(auto-fill,minmax(18rem,1fr))]">
{variants.map((variant) => (
<VariantCard
key={variant.profileId}
variant={variant}
profile={profiles.get(variant.profileId) ?? null}
navigate={navigate}
projectId={projectId}
onChanged={load}
/>
))}
</div>
</div>
);
}
function VariantCard({
variant,
profile,
navigate,
projectId,
onChanged,
}: {
variant: Variant;
profile: PlatformProfileSummary | null;
navigate: MediaViewProps["navigate"];
projectId: string;
onChanged: () => Promise<void>;
}) {
const { t } = useLingui();
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const { plan, draft, final } = variant;
// The final render once there is one, otherwise the draft.
const shown = final ?? draft;
const status = shown ? gateStatus(shown) : null;
const playable = shown?.status === "done" ? shown : null;
const videoUrl = useMediaObjectUrl(playable ? renderVideoUrl(playable) : null);
const posterUrl = useMediaObjectUrl(playable ? renderPosterUrl(playable) : null);
const fixRounds = shown?.gateReport?.autoFixRound ?? 0;
const finalVideo = final?.status === "done" ? renderVideoUrl(final) : null;
const draftApprovable =
plan !== null &&
draft?.status === "done" &&
draft.planId === plan.id &&
gateStatus(draft) !== "blocked" &&
!(final && (isRenderActive(final) || final.planId === plan.id));
async function act(work: () => Promise<unknown>) {
setBusy(true);
setError(null);
try {
await work();
await onChanged();
} catch (cause) {
setError(errorMessage(cause));
} finally {
setBusy(false);
}
}
const start = (quality: "draft" | "final") =>
plan ? act(() => rpc.media.renders.start({ planId: plan.id, quality })) : undefined;
return (
<Card data-testid="media-variant" data-profile={variant.profileId}>
<CardHeader className="flex flex-row items-center gap-2">
<CardTitle className="min-w-0 flex-1 truncate">
{profile?.name ?? variant.profileId}
</CardTitle>
{profile && (profile.unverified || profile.stale) ? (
<span
className="text-ui-caption text-muted-foreground"
title={t`Some limits of this profile are unverified or out of date.`}
>
<Trans>Unverified</Trans>
</span>
) : null}
{status ? <GateBadge status={status} /> : null}
</CardHeader>
<CardContent className="flex flex-col gap-3">
{shown && isRenderActive(shown) ? (
<div className="flex aspect-video items-center justify-center rounded-lg bg-muted">
<LoadingState
label={`${shown.status === "gating" ? t`Checking` : t`Rendering`} ${Math.round(shown.progress * 100)}%`}
startedAt={Date.parse(shown.createdAt)}
/>
</div>
) : playable ? (
<RenderVideo src={videoUrl} poster={posterUrl} label={t`Variant preview`} />
) : null}
{shown?.status === "failed" && shown.error ? (
<p className="text-ui-body-sm text-destructive">{shown.error}</p>
) : null}
<div className="flex items-center gap-2 text-ui-body-sm text-muted-foreground">
{plan ? <span>{formatMs(edlDurationMs(plan.edl))}</span> : <Trans>No plan yet</Trans>}
{fixRounds > 0 ? (
<span>
<Trans>Auto-fix {fixRounds}/3</Trans>
</span>
) : null}
</div>
<div className="flex flex-wrap gap-2">
{plan ? (
<Button
variant="outline"
size="sm"
onClick={() => navigate({ name: "editor", projectId, planId: plan.id })}
>
<Pencil />
<Trans>Edit</Trans>
</Button>
) : null}
{plan && !(draft && isRenderActive(draft)) ? (
<Button variant="outline" size="sm" disabled={busy} onClick={() => start("draft")}>
<Play />
<Trans>Draft</Trans>
</Button>
) : null}
{shown && isRenderActive(shown) ? (
<Button
variant="ghost"
size="sm"
disabled={busy}
onClick={() => act(() => rpc.media.renders.cancel({ renderId: shown.id }))}
>
<X />
<Trans>Cancel</Trans>
</Button>
) : null}
{shown?.gateReport ? (
<Button
variant="outline"
size="sm"
onClick={() => navigate({ name: "review", projectId, renderId: shown.id })}
>
<Trans>Review</Trans>
</Button>
) : null}
{draftApprovable ? (
<Button size="sm" disabled={busy} onClick={() => start("final")}>
<ShieldCheck />
<Trans>Approve</Trans>
</Button>
) : null}
{finalVideo ? (
<Button
variant="ghost"
size="icon-sm"
aria-label={t`Download`}
onClick={() =>
act(() => downloadMediaFile(`${finalVideo}?download=1`, `${variant.profileId}.mp4`))
}
>
<Download />
</Button>
) : null}
</div>
<ErrorLine error={error} />
{final?.status === "done" ? <PublishPanel render={final} /> : null}
</CardContent>
</Card>
);
}

View File

@ -0,0 +1,6 @@
import type { MediaViewProps } from "./nav";
/** Placeholder; the W3-web-library slice replaces this view. */
export function ResultsView(_props: MediaViewProps & Record<string, unknown>) {
return null;
}

View File

@ -0,0 +1,597 @@
import { Trans, useLingui } from "@lingui/react/macro";
import type {
AnonymizeMode,
Edl,
GateIssue,
MarkTreatment,
MediaCapabilities,
MediaEditPlan,
MediaMark,
MediaPerson,
MediaRender,
} from "@rakazo/contracts";
import { MAX_AUTO_FIX_ROUNDS } from "@rakazo/contracts";
import { Button, cn, NativeSelect, NativeSelectOption, Skeleton } from "@rakazo/ui-web";
import { Crosshair, Play, Plus, Save, Trash2 } from "lucide-react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { LoadingState } from "../../components/ai/primitives";
import {
ErrorLine,
errorMessage,
GateBadge,
ViewHeader,
} from "../../components/media/editor/chrome";
import {
renderPosterUrl,
renderVideoUrl,
useMediaObjectUrl,
usePolling,
} from "../../components/media/editor/media-files";
import type { PlayerHandle, SourceFrame } from "../../components/media/editor/Player";
import { RenderVideo, SourcePlayer } from "../../components/media/editor/Player";
import {
gateStatus,
isRenderActive,
RENDER_POLL_MS,
reviewedPlan,
sortedIssues,
} from "../../components/media/editor/renders";
import type { CoverageRow } from "../../components/media/editor/review-parts";
import { CoverageHeatmap, MaskBoxEditor } from "../../components/media/editor/review-parts";
import {
addRegion,
edlDurationMs,
formatMs,
regionBoxAt,
removeRegion,
setRegionKeyframe,
} from "../../components/media/editor/timeline";
import { rpc } from "../../lib/rpc";
import type { MediaViewProps } from "./nav";
type Mode = "after" | "before";
/**
* Review of one render (spec 7 Prüfansicht): gate blockers that jump the
* player, coverage per person and mark, before/after, manual masks, and
* per-person and per-mark treatment.
*/
export function ReviewView({
navigate,
projectId,
renderId,
}: MediaViewProps & { projectId: string; renderId: string }) {
const { t } = useLingui();
const [render, setRender] = useState<MediaRender | null>(null);
const [plans, setPlans] = useState<MediaEditPlan[]>([]);
const [edl, setEdl] = useState<Edl | null>(null);
const [savedPlan, setSavedPlan] = useState<MediaEditPlan | null>(null);
const [dirty, setDirty] = useState(false);
const [people, setPeople] = useState<MediaPerson[]>([]);
const [marks, setMarks] = useState<MediaMark[]>([]);
const [capabilities, setCapabilities] = useState<MediaCapabilities | null>(null);
const [mode, setMode] = useState<Mode>("after");
const [marking, setMarking] = useState(false);
const [selectedMask, setSelectedMask] = useState<string | null>(null);
const [frame, setFrame] = useState<SourceFrame | null>(null);
const [busy, setBusy] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
const renderPlayer = useRef<PlayerHandle>(null);
const sourcePlayer = useRef<PlayerHandle>(null);
const kindLabels = useMarkKindLabels();
const loadRender = useCallback(async () => {
try {
setRender(await rpc.media.renders.get({ renderId }));
} catch (cause) {
setError(errorMessage(cause));
}
}, [renderId]);
const loadMarks = useCallback(async () => {
setMarks(await rpc.media.marks.list({}));
}, []);
useEffect(() => {
setEdl(null);
setSavedPlan(null);
setDirty(false);
void loadRender();
void rpc.media.projects
.get({ projectId })
.then((data) => setPlans(data.plans))
.catch((cause) => setError(errorMessage(cause)));
void rpc.media.people
.list({})
.then(setPeople)
.catch(() => undefined);
void loadMarks().catch(() => undefined);
void rpc.media
.capabilities({})
.then(setCapabilities)
.catch(() => undefined);
}, [projectId, loadRender, loadMarks]);
usePolling(loadRender, render ? isRenderActive(render) : false, RENDER_POLL_MS);
const plan = useMemo(
() => savedPlan ?? (render ? reviewedPlan(render, plans) : null),
[savedPlan, render, plans],
);
useEffect(() => {
if (plan && !dirty) setEdl(plan.edl);
}, [plan, dirty]);
const videoUrl = useMediaObjectUrl(render?.status === "done" ? renderVideoUrl(render) : null);
const posterUrl = useMediaObjectUrl(render?.status === "done" ? renderPosterUrl(render) : null);
const hasVideo = render?.status === "done" && renderVideoUrl(render) !== null;
const shownMode: Mode = hasVideo ? mode : "before";
if (!render || !edl || !plan) {
return (
<div className="flex flex-col gap-4 p-4">
<ErrorLine error={error} />
{error ? null : <Skeleton className="h-64 w-full" />}
</div>
);
}
const issues = sortedIssues(render.gateReport?.issues ?? []);
const status = gateStatus(render);
const round = render.gateReport?.autoFixRound ?? 0;
const durationMs = edlDurationMs(edl);
const seek = (ms: number) => {
if (shownMode === "after") renderPlayer.current?.seek(ms);
else sourcePlayer.current?.seek(ms);
};
const changeEdl = (next: Edl) => {
setEdl(next);
setDirty(true);
};
async function act(key: string, work: () => Promise<void>) {
setBusy(key);
setError(null);
try {
await work();
} catch (cause) {
setError(errorMessage(cause));
} finally {
setBusy(null);
}
}
const savePlan = () =>
act("save", async () => {
const result = await rpc.media.plans.submit({ projectId, edl, rationale: null });
if (!result.ok) {
setError(result.errors.map((item) => item.message).join(" "));
return;
}
setSavedPlan(result.plan);
setPlans((current) => [...current, result.plan]);
setDirty(false);
});
const rerender = () =>
act("render", async () => {
const started = await rpc.media.renders.start({ planId: plan.id, quality: "draft" });
navigate({ name: "review", projectId, renderId: started.id });
});
// People in this video: face redactions by person plus anyone the gate named.
const personIds = new Set<string>();
for (const face of edl.redactions.faces) {
if ("personId" in face.target) personIds.add(face.target.personId);
}
for (const issue of issues) if (issue.personId) personIds.add(issue.personId);
const shownPeople = people.filter((person) => personIds.has(person.id));
const assetIds = new Set(edl.tracks.video.map((clip) => clip.assetId));
const plannedMarks = new Set(edl.redactions.marks.map((mark) => mark.markId));
const shownMarks = marks.filter(
(mark) =>
(mark.assetId !== null && assetIds.has(mark.assetId)) ||
(mark.personId !== null && personIds.has(mark.personId)) ||
plannedMarks.has(mark.id) ||
issues.some((issue) => issue.markId === mark.id),
);
const coverage: CoverageRow[] = [
...shownPeople.map((person) => ({
key: `person-${person.id}`,
label: person.displayName,
issues: issues.filter((issue) => issue.personId === person.id),
})),
...shownMarks.map((mark) => ({
key: `mark-${mark.id}`,
label: mark.label ?? kindLabels[mark.kind],
issues: issues.filter((issue) => issue.markId === mark.id),
})),
];
const unassigned = issues.filter(
(issue) => issue.check === "anonymization" && !issue.personId && !issue.markId,
);
if (unassigned.length > 0) {
coverage.push({ key: "unassigned", label: t`Unknown faces`, issues: unassigned });
}
const clipRegions = frame
? edl.redactions.regions.filter((region) => region.clipId === frame.clipId)
: [];
return (
<div className="flex flex-col gap-4 p-4" data-testid="media-review">
<ViewHeader
title={`${render.profileId} · ${render.quality === "final" ? t`Final` : t`Draft`}`}
backLabel={t`Back to project`}
onBack={() => navigate({ name: "project", projectId })}
>
{status ? <GateBadge status={status} /> : null}
</ViewHeader>
{isRenderActive(render) ? (
<LoadingState
label={`${render.status === "gating" ? t`Checking` : t`Rendering`} ${Math.round(render.progress * 100)}%`}
startedAt={Date.parse(render.createdAt)}
/>
) : null}
{round > 0 ? (
<p className="text-ui-body-sm text-muted-foreground" data-testid="media-autofix">
<Trans>
Auto-fix {round}/{MAX_AUTO_FIX_ROUNDS}
</Trans>
{plan.createdBy === "agent" && plan.rationale ? ` · ${plan.rationale}` : null}
</p>
) : null}
{render.status === "failed" && render.error ? (
<p className="text-ui-body-sm text-destructive">{render.error}</p>
) : null}
<div className="mx-auto flex w-full max-w-2xl flex-col gap-2">
{hasVideo ? (
<div className="flex gap-1 self-center rounded-lg bg-muted p-0.5">
{(["before", "after"] as const).map((value) => (
<Button
key={value}
size="xs"
variant={mode === value ? "secondary" : "ghost"}
aria-pressed={mode === value}
onClick={() => setMode(value)}
>
{value === "before" ? t`Before` : t`After`}
</Button>
))}
</div>
) : null}
{shownMode === "after" ? (
<RenderVideo
src={videoUrl}
poster={posterUrl}
handleRef={renderPlayer}
label={t`Rendered video`}
/>
) : (
<SourcePlayer
edl={edl}
handleRef={sourcePlayer}
onFrame={setFrame}
label={t`Source video`}
>
{frame
? clipRegions.map((region) => (
<MaskBoxEditor
key={region.id}
box={regionBoxAt(region, frame.sourceMs)}
label={t`Mask ${region.id}`}
selected={selectedMask === region.id}
onSelect={() => setSelectedMask(region.id)}
onChange={(box) =>
changeEdl(setRegionKeyframe(edl, region.id, frame.sourceMs, box))
}
/>
))
: null}
{marking && frame ? (
<button
type="button"
aria-label={t`Mark this spot`}
className="pointer-events-auto absolute inset-0 cursor-crosshair"
onClick={(event) => {
const box = event.currentTarget.getBoundingClientRect();
const point = {
x: Math.min(1, Math.max(0, (event.clientX - box.left) / box.width)),
y: Math.min(1, Math.max(0, (event.clientY - box.top) / box.height)),
};
setMarking(false);
void act("mark", async () => {
await rpc.media.marks.create({
kind: "custom",
label: null,
personId: null,
assetId: frame.assetId,
prompt: { tMs: frame.sourceMs, point },
treatment: "blur",
});
await loadMarks();
});
}}
/>
) : null}
</SourcePlayer>
)}
{shownMode === "before" ? (
<div className="flex flex-wrap items-center gap-2">
<Button
variant="outline"
size="sm"
disabled={!frame}
onClick={() => {
if (!frame) return;
const added = addRegion(edl, frame.clipId, frame.sourceMs);
changeEdl(added.edl);
setSelectedMask(added.id);
}}
>
<Plus />
<Trans>Mask</Trans>
</Button>
<Button
variant={marking ? "secondary" : "outline"}
size="sm"
aria-pressed={marking}
disabled={!frame || busy === "mark"}
onClick={() => setMarking((value) => !value)}
>
{busy === "mark" ? (
<LoadingState label={t`Marking`} />
) : (
<>
<Crosshair />
<Trans>Mark</Trans>
</>
)}
</Button>
{selectedMask && edl.redactions.regions.some((r) => r.id === selectedMask) ? (
<Button
variant="ghost"
size="sm"
onClick={() => {
changeEdl(removeRegion(edl, selectedMask));
setSelectedMask(null);
}}
>
<Trash2 />
<Trans>Remove mask</Trans>
</Button>
) : null}
</div>
) : null}
{dirty ? (
<Button size="sm" className="self-start" disabled={busy !== null} onClick={savePlan}>
<Save />
<Trans>Save</Trans>
</Button>
) : savedPlan ? (
<Button size="sm" className="self-start" disabled={busy !== null} onClick={rerender}>
<Play />
<Trans>Draft</Trans>
</Button>
) : null}
<ErrorLine error={error} />
</div>
<CoverageHeatmap rows={coverage} durationMs={durationMs} onSeek={seek} />
{issues.length > 0 ? (
<ol className="flex flex-col divide-y divide-border rounded-lg border border-border">
{issues.map((issue, index) => (
<IssueRow
key={`${issue.code}-${issue.atMs ?? "x"}-${index}`}
issue={issue}
onSeek={seek}
/>
))}
</ol>
) : null}
{shownPeople.length > 0 ? (
<section className="flex flex-col gap-2">
{shownPeople.map((person) => (
<PersonRow
key={person.id}
person={person}
syntheticAvailable={capabilities?.operations.synthesizeIdentity?.available === true}
needsConsent={issues.some(
(issue) => issue.check === "consent" && issue.personId === person.id,
)}
onConsent={() => navigate({ name: "people" })}
onChange={(next) =>
setPeople((current) => current.map((p) => (p.id === next.id ? next : p)))
}
/>
))}
</section>
) : null}
{shownMarks.length > 0 ? (
<section className="flex flex-col gap-2">
{shownMarks.map((mark) => (
<MarkRow
key={mark.id}
mark={mark}
onChange={(next) =>
setMarks((current) => current.map((m) => (m.id === next.id ? next : m)))
}
/>
))}
</section>
) : null}
</div>
);
}
function IssueRow({ issue, onSeek }: { issue: GateIssue; onSeek: (ms: number) => void }) {
const { t } = useLingui();
const timed = issue.atMs !== undefined;
return (
<li data-testid="media-gate-issue" data-severity={issue.severity}>
<button
type="button"
disabled={!timed}
onClick={() => timed && onSeek(issue.atMs ?? 0)}
className="flex w-full items-start gap-3 px-3 py-2 text-left enabled:hover:bg-muted/50"
>
<span
aria-hidden
className={cn(
"mt-1.5 size-2 shrink-0 rounded-full",
issue.severity === "block" ? "bg-destructive" : "bg-warning",
)}
/>
<span className="w-14 shrink-0 font-mono text-ui-body-sm tabular-nums text-muted-foreground">
{timed ? formatMs(issue.atMs ?? 0) : "—"}
</span>
<span className="min-w-0 flex-1 text-ui-body-sm">
<span className="sr-only">{issue.severity === "block" ? t`Blocker` : t`Warning`}: </span>
{issue.message}
</span>
</button>
</li>
);
}
function PersonRow({
person,
syntheticAvailable,
needsConsent,
onConsent,
onChange,
}: {
person: MediaPerson;
syntheticAvailable: boolean;
needsConsent: boolean;
onConsent: () => void;
onChange: (person: MediaPerson) => void;
}) {
const { t } = useLingui();
const [error, setError] = useState<string | null>(null);
const modes: Array<{ value: AnonymizeMode; label: string }> = [
{ value: "none", label: t`Visible` },
{ value: "blur", label: t`Blur` },
{ value: "pixelate", label: t`Pixelate` },
{ value: "solid", label: t`Solid` },
{ value: "sticker", label: t`Sticker` },
{ value: "partial_eyes", label: t`Eyes` },
{ value: "partial_upper", label: t`Upper face` },
{ value: "partial_lower", label: t`Lower face` },
{ value: "reframe_out", label: t`Crop out` },
...(syntheticAvailable || person.anonymizeByDefault === "synthetic_identity"
? [{ value: "synthetic_identity" as const, label: t`Synthetic face (experimental)` }]
: []),
];
return (
<div className="flex flex-wrap items-center gap-2" data-testid="media-review-person">
<span className="min-w-0 flex-1 truncate text-ui-body">{person.displayName}</span>
{needsConsent ? (
<Button variant="outline" size="sm" onClick={onConsent}>
<Trans>Record consent</Trans>
</Button>
) : null}
<NativeSelect
size="sm"
aria-label={t`Anonymization for ${person.displayName}`}
value={person.anonymizeByDefault}
onChange={(event) => {
const anonymizeByDefault = event.target.value as AnonymizeMode;
setError(null);
void rpc.media.people
.update({ personId: person.id, anonymizeByDefault })
.then(onChange)
.catch((cause) => setError(errorMessage(cause)));
}}
>
{modes.map((mode) => (
<NativeSelectOption key={mode.value} value={mode.value}>
{mode.label}
</NativeSelectOption>
))}
</NativeSelect>
<ErrorLine error={error} />
</div>
);
}
function MarkRow({ mark, onChange }: { mark: MediaMark; onChange: (mark: MediaMark) => void }) {
const { t } = useLingui();
const [error, setError] = useState<string | null>(null);
const labels = useTreatmentLabels();
const kindLabels = useMarkKindLabels();
const update = (treatment: MarkTreatment) => {
setError(null);
void rpc.media.marks
.update({ markId: mark.id, treatment })
.then(onChange)
.catch((cause) => setError(errorMessage(cause)));
};
const fallback =
mark.suggestedFallback && mark.suggestedFallback !== mark.treatment
? mark.suggestedFallback
: null;
return (
<div className="flex flex-wrap items-center gap-2" data-testid="media-review-mark">
<span className="min-w-0 flex-1 truncate text-ui-body">
{mark.label ?? kindLabels[mark.kind]}
</span>
{mark.qualityScore !== null ? (
<span className="font-mono text-ui-body-sm tabular-nums text-muted-foreground">
{Math.round(mark.qualityScore * 100)}%
</span>
) : null}
{fallback ? (
<Button variant="outline" size="sm" onClick={() => update(fallback)}>
{t`Use ${labels[fallback]}`}
</Button>
) : null}
<NativeSelect
size="sm"
aria-label={t`Treatment`}
value={mark.treatment}
onChange={(event) => update(event.target.value as MarkTreatment)}
>
{(Object.keys(labels) as MarkTreatment[]).map((value) => (
<NativeSelectOption key={value} value={value}>
{labels[value]}
</NativeSelectOption>
))}
</NativeSelect>
<ErrorLine error={error} />
</div>
);
}
function useTreatmentLabels(): Record<MarkTreatment, string> {
const { t } = useLingui();
return {
inpaint: t`Remove`,
blur: t`Blur`,
cover: t`Cover`,
reframe_out: t`Crop out`,
none: t`Keep`,
};
}
function useMarkKindLabels(): Record<MediaMark["kind"], string> {
const { t } = useLingui();
return {
tattoo: t`Tattoo`,
birthmark: t`Birthmark`,
scar: t`Scar`,
jewelry: t`Jewelry`,
background_detail: t`Background detail`,
license_plate: t`License plate`,
custom: t`Marked spot`,
};
}

View File

@ -0,0 +1,14 @@
/** Screens inside Media Studio. Every view receives `navigate` to move between them. */
export type MediaView =
| { name: "library" }
| { name: "collection"; collectionId: string }
| { name: "people" }
| { name: "results" }
| { name: "project"; projectId: string }
| { name: "editor"; projectId: string; planId: string }
| { name: "review"; projectId: string; renderId: string }
| { name: "brandKit"; brandKitId?: string };
export interface MediaViewProps {
navigate: (view: MediaView) => void;
}

View File

@ -38,10 +38,13 @@ export type NewSpaceConfirmInput = {
export function NewSpaceDialog({
variant = "space",
canLinkFolder,
onCancel,
onConfirm,
}: {
variant?: "space" | "project";
/** Folders live on the host, which only the deployment owner may link. */
canLinkFolder: boolean;
onCancel: () => void;
onConfirm: (input: NewSpaceConfirmInput) => Promise<void>;
}) {
@ -53,6 +56,7 @@ export function NewSpaceDialog({
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const isProject = variant === "project";
const showFolder = isProject && canLinkFolder;
const desktop = desktopBridge();
const trimmedWorkspacePath = workspacePath.trim();
@ -63,7 +67,7 @@ export function NewSpaceDialog({
setError(null);
void onConfirm({
name: trimmed,
...(isProject && trimmedWorkspacePath ? { workspacePath: trimmedWorkspacePath } : {}),
...(showFolder && trimmedWorkspacePath ? { workspacePath: trimmedWorkspacePath } : {}),
}).catch((reason: unknown) => {
setError(
reason instanceof Error
@ -104,7 +108,7 @@ export function NewSpaceDialog({
className="mt-2"
/>
</label>
{isProject ? (
{showFolder ? (
<div className="space-y-2">
<label htmlFor={folderId} className="block text-ui-body text-foreground/75">
<Trans>Project folder</Trans>

View File

@ -240,6 +240,26 @@ body,
-webkit-app-region: no-drag;
}
/* In-app browser load indicator: a short bar sweeping under the address bar. */
.rk-browser-progress {
animation: rkBrowserProgress 1.1s cubic-bezier(0.65, 0, 0.35, 1) infinite;
}
@keyframes rkBrowserProgress {
from {
transform: translateX(-100%);
}
to {
transform: translateX(300%);
}
}
@media (prefers-reduced-motion: reduce) {
.rk-browser-progress {
animation: none;
width: 100%;
opacity: 0.6;
}
}
@keyframes rkPulse {
0%,
100% {

View File

@ -10,6 +10,8 @@ import {
createCloudAgentConnection,
createConnectorStack,
createJobReconciler,
createMediaJobHandlers,
createMediaServices,
createMessagingContextLoader,
createPostgresReconciliationLeadership,
createProfileConsolidationScheduler,
@ -31,6 +33,7 @@ import {
LocalArtifactStore,
McpConnector,
McpOAuthBroker,
mediaConfigFromEnv,
messagingEnvFromProcess,
messagingPlatformsFromEnv,
PiAgentRuntime,
@ -201,6 +204,25 @@ async function main() {
cloudAgent,
});
const mediaAbort = new AbortController();
const mediaServices = createMediaServices({
config: mediaConfigFromEnv(process.env),
prisma,
jobs,
artifacts,
runtime,
realtime,
resolveModel: (scope) => executor.resolveModel(scope),
});
const mediaJobs = mediaServices
? createMediaJobHandlers(mediaServices, {
signal: mediaAbort.signal,
limits: {
gpu: parsePositiveInteger(process.env.MEDIA_GPU_CONCURRENCY, 1),
cpu: parsePositiveInteger(process.env.MEDIA_CPU_CONCURRENCY, 2),
},
})
: null;
const jobHandlers = createBackgroundJobHandlers({
executor,
prisma,
@ -208,6 +230,7 @@ async function main() {
home,
jobs,
events,
...(mediaJobs ? { media: mediaJobs.handlers } : {}),
workerId: process.pid.toString(),
runtime,
secretStore: secrets,
@ -235,6 +258,9 @@ async function main() {
await new Promise((resolve) => setTimeout(resolve, databaseCapacityBackoffMs(attempt)));
}
}
if (mediaJobs) {
void mediaJobs.resume().catch((error) => logger.error("media ingest resume failed", error));
}
const reconciler = createJobReconciler({
prisma,
jobs,
@ -253,6 +279,7 @@ async function main() {
stopping = true;
try {
await reconciler.stop();
mediaAbort.abort();
await jobHost.stop();
await jobs.close();
await realtime.close();

View File

@ -129,6 +129,28 @@ services:
data-init:
condition: service_completed_successfully
media-worker:
# Optional Media Studio sidecar: `docker compose --profile media up`.
# No published port; the API and worker reach it as http://media-worker:8790.
profiles: ["media"]
build:
context: ../media-worker
args:
ACCELERATOR: ${MEDIA_ACCELERATOR:-cpu}
image: rakazo/media-worker:local
environment:
MEDIA_WORKER_HOST: "0.0.0.0"
MEDIA_WORKER_PORT: "8790"
MEDIA_WORKER_TOKEN: ${MEDIA_WORKER_TOKEN:?Set MEDIA_WORKER_TOKEN in .env}
MEDIA_ROOT: /data/media
MEDIA_MODELS_DIR: /models
HTTP_PROXY: ${HTTP_PROXY:-${http_proxy:-}}
HTTPS_PROXY: ${HTTPS_PROXY:-${https_proxy:-}}
NO_PROXY: ${NO_PROXY:-${no_proxy:-}}
volumes:
- ../../data/media:/data/media
- media-models:/models
web:
build:
context: ../..
@ -144,4 +166,5 @@ services:
- api
volumes:
media-models:
pgdata:

View File

@ -0,0 +1,7 @@
.pytest_cache
.ruff_cache
.mypy_cache
.venv
.venv-ml
**/__pycache__
**/*.pyc

5
infra/media-worker/.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
.pytest_cache/
.venv/
.venv
.venv-ml
__pycache__/

View File

@ -0,0 +1,68 @@
# syntax=docker/dockerfile:1.7
# Media Studio inference sidecar.
# docker build --build-arg ACCELERATOR=cpu|cuda|rocm -t rakazo/media-worker infra/media-worker
# docker build --target test -t rakazo/media-worker:test infra/media-worker # offline pytest
# Model weights are not baked in; they download into /models on first use.
# Behind a TLS-intercepting proxy, pass its CA with --secret id=pip_ca,src=<ca.pem>.
ARG ACCELERATOR=cpu
FROM python:3.11-slim-bookworm@sha256:2333bd330d12de02514770b3585cad313644316047cdee24a7acfdece6de6efb AS runtime
ARG ACCELERATOR
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
TQDM_DISABLE=1 \
MEDIA_ACCELERATOR=${ACCELERATOR} \
MEDIA_MODELS_DIR=/models \
MEDIA_WORKER_PORT=8790
RUN case "${ACCELERATOR}" in cpu|cuda|rocm) ;; *) echo "ACCELERATOR must be cpu, cuda or rocm" >&2; exit 1 ;; esac \
&& apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg libsndfile1 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /srv/media-worker
COPY requirements.txt requirements-ml-cpu.txt requirements-ml-cuda.txt requirements-ml-rocm.txt ./
# faster-whisper depends on the CPU onnxruntime wheel; the CUDA image swaps in
# onnxruntime-gpu. SAM2 is an sdist built against the installed torch, without
# its optional CUDA extension.
RUN --mount=type=secret,id=pip_ca,required=false \
if [ -s /run/secrets/pip_ca ]; then export PIP_CERT=/run/secrets/pip_ca; fi \
&& pip install -r requirements.txt -r "requirements-ml-${ACCELERATOR}.txt" \
&& if [ "${ACCELERATOR}" = "cuda" ]; then \
pip uninstall -y onnxruntime && pip install --no-deps --force-reinstall onnxruntime-gpu==1.30.0; \
fi \
&& SAM2_BUILD_CUDA=0 pip install --no-build-isolation --no-deps sam2==1.1.0 \
&& python -c "import cv2, scenedetect, librosa, onnxruntime, faster_whisper, torch, sam2, tokenizers"
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin media \
&& mkdir -p /models \
&& chown media:media /models
VOLUME ["/models"]
COPY app ./app
USER media
EXPOSE 8790
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD ["python", "-c", "import os, urllib.request; urllib.request.urlopen('http://127.0.0.1:' + os.environ.get('MEDIA_WORKER_PORT', '8790') + '/healthz', timeout=4)"]
CMD ["python", "-m", "app"]
# Offline test image: same runtime plus pytest; fake backends, no weights.
FROM runtime AS test
USER root
COPY requirements-dev.txt pytest.ini ./
RUN --mount=type=secret,id=pip_ca,required=false \
if [ -s /run/secrets/pip_ca ]; then export PIP_CERT=/run/secrets/pip_ca; fi \
&& pip install -r requirements-dev.txt
COPY tests ./tests
USER media
CMD ["python", "-m", "pytest", "-q"]
FROM runtime

Some files were not shown because too many files have changed in this diff Show More