Compare commits

...

24 Commits

Author SHA1 Message Date
Claude
8f38c22119
wip(media): w2-render partial work (handover, unverified)
Stopped mid-implementation for the handover to the project thread.
Not typechecked or tested; do not merge as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:24:51 +00:00
Claude
2556ed197e
feat(media): service contracts between media slices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:11:04 +00:00
Claude
475710e27e
feat(media): shared service deps and consent-gated model client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:10:02 +00:00
Claude
d43e374534
merge: W1-domain media domain logic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:39 +00:00
Claude
a6598f09d9
test(media): cover clustering, EDL, compiler, teaser, gate and marketing
Includes compiler snapshots, injection-proof escaping checks, and an
ffmpeg integration test on synthetic testsrc/sine inputs that is skipped
when ffmpeg is not installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
26d95a374e
feat(media): implement pure domain logic for Media Studio
Session and theme clustering, EDL validation and timeline mapping, the
EDL-to-ffmpeg compiler with caption files and filtergraph escaping,
teaser selection and learning, the compliance gate with compliance-only
auto-fix, and the marketing helpers (UTM, hashtags, CSV metrics, intro
proposals, brand-kit checks, upload checklist).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:06:02 +00:00
Claude
1aab5ab3a8
feat(media): expose asset externalId, filter by ingest state, push device deletions
The desktop Photos sync needs to map awaiting assets back to Photos items,
query only awaiting uploads, and report deletions without a full relisting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 18:00:02 +00:00
Claude
51445e9442
merge: W1-desktop Apple Photos helper and sync
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:20 +00:00
Claude
0ee6c664d2
build(db): depend on @rakazo/media for coordinate rounding
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:57:02 +00:00
Claude
c877343218
merge: W1-db media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:32 +00:00
Claude
d3c6f262ad
feat(media): Apple Photos helper and client-pushed desktop sync
Adds the macOS photos-helper Swift CLI (PhotoKit listing with stable
cursors, persistent change tokens, iCloud-aware original export with a
path guard and SHA-256) plus XCTests for its PhotoKit-free core, and the
Electron main-process side: a validated helper client, a resumable
metadata push and chunked original upload against the media contract,
and IPC handlers that stay unavailable off macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:56:14 +00:00
Claude
5e155a0850
test(media): Postgres and unit tests for the media repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c1c39b6e46
feat(media): space-scoped Media Studio repositories
Sources, assets with keyset search, analyses, segments, uploads,
collections with re-clustering, people with consent and complete
biometric removal, marks, brand kits, projects with versioned plans,
renders with atomic claims, publications, metrics and the funnel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:55:49 +00:00
Claude
c3f06e4a12
feat(media): wire media adapters and ffmpeg for CI media tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:53:43 +00:00
Claude
0c4e0202f3
merge: W1-adapters emulator, sidecar client, folder source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:59 +00:00
Claude
270ea8068a
feat(media): inference emulator, sidecar client, folder source and factory
Add the Media Studio adapters behind the frozen provider contracts: a
deterministic offline inference emulator driven by file-name markers that
renders real redactions with ffmpeg, a synthetic emulator library, the
bearer-auth HTTP client for the local media sidecar with validated
responses, typed errors, busy retries and timeouts, a lazily paginated
folder source that never follows symlinks, and the configuration factory.
A shared conformance suite runs against the emulator and, in the Docker
integration job, against the sidecar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:52:17 +00:00
Claude
f98fd9703c
feat(media): export motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:48:29 +00:00
Claude
759862fef4
merge: W1-motion motion graphics renderer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:57 +00:00
Claude
b5cef21f44
feat(media): resvg motion graphics renderer with brand-kit templates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:47:24 +00:00
Claude
7ccc96ac11
feat(media): shared geo, path-safety and profile loading helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:34:04 +00:00
Claude
a663764ee8
feat(media): contracts and foundation for Media Studio
Freeze the shared contracts every Media Studio slice builds on: zod DTOs,
EDL, platform profile, brand kit, gate and render plan schemas; the media
RPC surface with a scaffold router; provider interfaces for sources,
inference, rendering, motion graphics and publishing; media background
jobs; the Prisma models and migration; and the @rakazo/media domain
package signatures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138YqDj3ttfayRwuMKKrqVj
2026-10-03 17:32:52 +00:00
cynarAI
3c3961f242
fix(api): keep host folders to the deployment owner (#112)
spaces.create linked any workspacePath to the new project, and
computer.files and computer.readFile then listed and read files under
that folder for anyone in the space. On a deployment with signups, any
member could link / or the API's own checkout and read whatever the API
process can read, such as its .env file. Linking also ran git in the
chosen folder and stored its remote URL.

A linked folder lives on the host, and the host is the deployment
owner's machine, the rule the onboarding scan already follows:

- spaces.create refuses a folder from anyone but the deployment owner,
  before the space is created or the folder is touched.
- Project files are listed and read only for the deployment owner, so a
  space a member linked before this change stays closed too. Space
  navigation no longer shows a linked folder's path and git remote to
  other members.
- A linked folder is stored as its real path, and listing and reading
  follow symlinks only while they stay inside it. Symlinks that leave
  the folder or lead nowhere are left out of a listing instead of
  failing it.
- The web New project dialog offers the folder field only to the
  deployment owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EyQw3AEaHYXegvgvTPAsi1
2026-10-03 18:32:48 +02:00
cynarAI
2b967df569
fix(api): turn repositories found by the onboarding scan into real projects (#116)
The scan's project phase called prisma.space with fields the Space model
does not have (title, path). Both calls threw, a per-repository
try/catch swallowed the errors, and no project was ever created. With
the field names fixed, the space would still have had no organization
or membership, so nobody could have seen it.

The phase now goes through createSpaceForMember and
wireProjectWorkspace, the helpers behind spaces.create: each repository
becomes a project of the deployment owner in their current
organization, linked to its folder with its git remote, branch and
head. It still runs only in the owner's scan with ONBOARDING_SCAN_HOME
set. A repository whose real folder one of the owner's projects already
links is skipped, so scanning again creates nothing; the phase stops at
the per-member space limit; and a second start while a scan runs (a
StrictMode remount, a retry) joins it instead of linking everything
twice. Discovery no longer descends into dot folders, where shell and
editor plugin checkouts would otherwise become projects. The Prisma
client is typed, so a field the schema lacks is a compile error again
instead of a swallowed one.

readGitCheckout loaded child_process with require(), which does not
exist in the ES module runtime the API runs in, so linking any folder
with a .git directory threw "require is not defined", in the scan and
in the New project dialog alike. It now imports spawn statically; the
module is only reachable through its server-only subpath, so browser
bundles still never load it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ebcq9seT9eCoHtqxuWEUzg
2026-10-03 18:22:10 +02:00
cynarAI
f8eabee997
fix(web): keep secondary shell text readable in the light theme (#117)
Secondary shell text was muted text dimmed further with opacity: the
section labels, the projects caption, the sidebar "+" icons, the privacy
panel, the trading subtitle and the Heute brief's Mission Control link.
Muted text measures 4.4:1 on the light sidebar, so these fell to 1.9 to
3.4:1 in the light theme; a few were below 4.5:1 in dark as well.

- Secondary sidebar text and icons share `rk-sidebar-muted`, the sidebar
  foreground at 65% that row times already used (5.0:1 on the light
  sidebar). Row times now use the class too.
- Section labels and space or project group titles use the dimmed
  foreground in light and keep the 80% muted color in dark; group titles
  move up from 60%, which was 3.4:1 in dark.
- The privacy panel heading is plain text with a success-colored check
  instead of emerald text, and its details pass 4.5:1 in both themes.
- Needs-you chips keep the warning tint and border; their label uses the
  foreground color in light.
- The Mission Control link uses the full muted color.
- E2E: the shell test checks every text in the sidebar except the logo,
  and the Heute brief, in both themes, and captures
  `shell-contrast-{light,dark}-heute`.
- AGENTS.md: agents make routine calls themselves, verify the result and
  report it, and ask only before irreversible actions or for product
  decisions without a sensible default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSVQGR4WU92LaddTP19GcQ
2026-10-03 17:56:44 +02:00
130 changed files with 29233 additions and 157 deletions

View File

@ -236,3 +236,10 @@ EVOO_COCKPIT_CORS_ORIGIN=
# Expo: English or Simplified Chinese (en | zh-CN). Overridden by SecureStore key
# rakazo.uiLocale from Account → Language.
# EXPO_PUBLIC_DEFAULT_UI_LOCALE=en
# Optional Media Studio worker (local ffmpeg + ML sidecar, compose profile "media").
# Without it, Media Studio stays hidden and media jobs are skipped.
# MEDIA_WORKER_URL=http://127.0.0.1:8790
# MEDIA_WORKER_TOKEN=REPLACE_WITH_RANDOM_TOKEN
# Private platform profiles that override the shipped ones. Keep outside the repo.
# MEDIA_PROFILES_DIR=~/.evoo/media-profiles

View File

@ -170,6 +170,15 @@ jobs:
- run: pnpm db:generate
env:
DATABASE_URL: postgres://rakazo:rakazo@127.0.0.1:5433/rakazo
- name: Ensure ffmpeg for media tests
shell: bash
run: |
if command -v ffmpeg >/dev/null && command -v ffprobe >/dev/null; then exit 0; fi
if sudo -n true 2>/dev/null; then
sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
else
echo "::warning::ffmpeg is missing on this runner; media render tests will be skipped"
fi
- run: pnpm test
test-integration:

View File

@ -10,6 +10,7 @@
- Keep code simple: reuse existing primitives and one source of truth, remove duplication and unused flexibility, and avoid speculative abstractions. Add an interface when it protects a real external or platform boundary, not for its own sake.
- Treat auth, secret handling, sandbox boundaries, host commands, and integrations as security-sensitive. Keep tests deterministic and offline by default.
- The desktop Playwright suite (`pnpm --filter @rakazo/desktop test:e2e`) opens real Electron windows and steals focus on macOS. Do not run it on a maintainer's machine as routine verification; run the unit tests locally and let CI's virtual-display job run the e2e on push.
- Work autonomously. Make routine calls yourself, such as opening or reopening a pull request, linking CI screenshots, or fixing gaps you found that belong to the task, then verify and report the outcome. Ask the maintainer only before irreversible actions or for product decisions without a sensible default.
- After creating a pull request, stay with it until CI and automated review bots have finished; passing checks alone do not mean the review is complete. Follow `.agents/skills/pr-watch/SKILL.md`: run its `pr-digest --watch` helper as a single backgrounded call that blocks until the checks on the current head commit are terminal, then act on the verdict it prints. Do not poll in the foreground on a timer, which mostly buys repeated snapshots of an unchanged PR and dies at the tool timeout. Address every actionable issue, push the fixes, and repeat until no actionable feedback remains. Do not merge while review bots are still pending or review issues remain unresolved.
- PR descriptions say why the change exists, what changed, and how it was tested. Write Why as the product reason (what was wrong and what this changes); never "Elie wants" or a third-person briefing of a named person; the maintainer is the author.
- Commit messages, PR descriptions, issues, and review replies are public too. Describe test results in words instead of pasting tool output, and never include anything that identifies a person, machine, or account: local paths, usernames, hostnames, signing identities, legal entity names, Apple Team IDs, API key IDs, issuer or tenant IDs, account emails, or the accounts, machines, and files that secret values came from. Naming the secrets a workflow reads is fine; use placeholders when a value must be shown.

View File

@ -19,6 +19,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/db": "workspace:*",
"@rakazo/logging": "workspace:*",
"@rakazo/media": "workspace:*",
"@rakazo/memory": "workspace:*",
"hono": "4.13.7"
},

113
apps/api/src/media.ts Normal file
View File

@ -0,0 +1,113 @@
import { implement, ORPCError } from "@orpc/server";
import type { Actor } from "@rakazo/contracts";
import { appContract } from "@rakazo/contracts";
import type { PrismaClient } from "@rakazo/db";
export interface MediaRouterDeps {
prisma: PrismaClient;
dataDir: string;
}
// Phase A scaffold: every procedure is replaced by its real implementation.
function notImplemented(): never {
throw new ORPCError("NOT_IMPLEMENTED", { message: "Media Studio is not available yet" });
}
export function createMediaRouter(_deps: MediaRouterDeps) {
const os = implement(appContract.media).$context<{ actor: Actor | null; signal?: AbortSignal }>();
const authed = os.use(async ({ context, next }) => {
if (!context.actor) throw new ORPCError("UNAUTHORIZED");
return next({ context: { ...context, actor: context.actor } });
});
return os.router({
capabilities: authed.capabilities.handler(notImplemented),
sources: {
list: authed.sources.list.handler(notImplemented),
createFolder: authed.sources.createFolder.handler(notImplemented),
registerClient: authed.sources.registerClient.handler(notImplemented),
pushItems: authed.sources.pushItems.handler(notImplemented),
sync: authed.sources.sync.handler(notImplemented),
remove: authed.sources.remove.handler(notImplemented),
},
assets: {
search: authed.assets.search.handler(notImplemented),
get: authed.assets.get.handler(notImplemented),
setPaidOnly: authed.assets.setPaidOnly.handler(notImplemented),
requestUpload: authed.assets.requestUpload.handler(notImplemented),
},
uploads: {
begin: authed.uploads.begin.handler(notImplemented),
status: authed.uploads.status.handler(notImplemented),
complete: authed.uploads.complete.handler(notImplemented),
},
collections: {
list: authed.collections.list.handler(notImplemented),
get: authed.collections.get.handler(notImplemented),
rename: authed.collections.rename.handler(notImplemented),
merge: authed.collections.merge.handler(notImplemented),
split: authed.collections.split.handler(notImplemented),
createManual: authed.collections.createManual.handler(notImplemented),
recluster: authed.collections.recluster.handler(notImplemented),
},
people: {
list: authed.people.list.handler(notImplemented),
update: authed.people.update.handler(notImplemented),
merge: authed.people.merge.handler(notImplemented),
assignTrack: authed.people.assignTrack.handler(notImplemented),
recordConsent: authed.people.recordConsent.handler(notImplemented),
remove: authed.people.remove.handler(notImplemented),
},
marks: {
list: authed.marks.list.handler(notImplemented),
create: authed.marks.create.handler(notImplemented),
update: authed.marks.update.handler(notImplemented),
remove: authed.marks.remove.handler(notImplemented),
},
brandKits: {
list: authed.brandKits.list.handler(notImplemented),
upsert: authed.brandKits.upsert.handler(notImplemented),
remove: authed.brandKits.remove.handler(notImplemented),
templates: authed.brandKits.templates.handler(notImplemented),
introVariants: authed.brandKits.introVariants.handler(notImplemented),
},
profiles: {
list: authed.profiles.list.handler(notImplemented),
get: authed.profiles.get.handler(notImplemented),
},
projects: {
list: authed.projects.list.handler(notImplemented),
get: authed.projects.get.handler(notImplemented),
create: authed.projects.create.handler(notImplemented),
update: authed.projects.update.handler(notImplemented),
remove: authed.projects.remove.handler(notImplemented),
autoPlan: authed.projects.autoPlan.handler(notImplemented),
},
plans: {
submit: authed.plans.submit.handler(notImplemented),
},
renders: {
start: authed.renders.start.handler(notImplemented),
get: authed.renders.get.handler(notImplemented),
cancel: authed.renders.cancel.handler(notImplemented),
thumbnails: authed.renders.thumbnails.handler(notImplemented),
},
marketing: {
copy: authed.marketing.copy.handler(notImplemented),
},
publications: {
list: authed.publications.list.handler(notImplemented),
create: authed.publications.create.handler(notImplemented),
bundle: authed.publications.bundle.handler(notImplemented),
markPublished: authed.publications.markPublished.handler(notImplemented),
cancel: authed.publications.cancel.handler(notImplemented),
},
metrics: {
mappings: authed.metrics.mappings.handler(notImplemented),
saveMapping: authed.metrics.saveMapping.handler(notImplemented),
importCsv: authed.metrics.importCsv.handler(notImplemented),
record: authed.metrics.record.handler(notImplemented),
funnel: authed.metrics.funnel.handler(notImplemented),
},
});
}

View File

@ -1,11 +1,42 @@
import { mkdir, mkdtemp, writeFile } from "node:fs/promises";
import { mkdir, mkdtemp, realpath, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type * as db from "@rakazo/db";
import { SpaceLimitError } from "@rakazo/db";
import { afterEach, describe, expect, it, vi } from "vitest";
import { getScanStatus, startOnboardingScan } from "./onboarding-scan.js";
const owner = { userId: "user-owner", isDeploymentOwner: true };
const member = { userId: "user-member", isDeploymentOwner: false };
const createSpaceForMember = vi.hoisted(() => vi.fn());
vi.mock("@rakazo/db", async (original) => ({
...(await original<typeof db>()),
createSpaceForMember,
}));
const owner = { userId: "user-owner", spaceId: "space-owner", isDeploymentOwner: true };
const member = { userId: "user-member", spaceId: "space-member", isDeploymentOwner: false };
type SpaceUpdate = { where: { id: string }; data: { workspacePath: string } };
/** The spaces the scan reads and links; new ones come from `createSpaceForMember`. */
function fakeDatabase({ spaceLimit = Number.POSITIVE_INFINITY } = {}) {
const spaces: Array<{ id: string; name: string; workspacePath: string | null }> = [];
createSpaceForMember.mockImplementation(async (_prisma: unknown, input: { name: string }) => {
if (spaces.length >= spaceLimit) throw new SpaceLimitError();
const space = { id: `space-${spaces.length + 1}`, name: input.name, workspacePath: null };
spaces.push(space);
return { id: space.id, name: space.name };
});
const prisma = {
space: {
findMany: async () => spaces.filter((space) => space.workspacePath),
update: async ({ where, data }: SpaceUpdate) => {
const space = spaces.find((candidate) => candidate.id === where.id);
if (space) space.workspacePath = data.workspacePath;
},
},
} as unknown as db.PrismaClient;
return { prisma, spaces };
}
async function scanHomeWithOneFile(): Promise<string> {
const home = await mkdtemp(join(tmpdir(), "evoo-scan-"));
@ -14,6 +45,12 @@ async function scanHomeWithOneFile(): Promise<string> {
return home;
}
async function scanHomeWithRepos(...repos: string[]): Promise<string> {
const home = await scanHomeWithOneFile();
for (const repo of repos) await mkdir(join(home, repo, ".git"), { recursive: true });
return home;
}
async function completed(scanId: string, userId: string) {
return await vi.waitFor(
async () => {
@ -25,13 +62,21 @@ async function completed(scanId: string, userId: string) {
);
}
/** The emulated runtime that the shared test setup pins keeps the scan simulated. */
function allowHostScan() {
vi.stubEnv("AGENT_RUNTIME", "");
vi.stubEnv("SANDBOX_PROVIDER", "");
}
describe("onboarding scan", () => {
afterEach(() => {
vi.unstubAllEnvs();
createSpaceForMember.mockReset();
});
it("hides a scan from every account except the one that started it", async () => {
const { scanId } = await startOnboardingScan({ actor: member }, {});
const { prisma } = fakeDatabase();
const { scanId } = await startOnboardingScan({ actor: member }, prisma);
await expect(getScanStatus(scanId, owner.userId)).rejects.toThrow(/Scan not found/);
await expect(getScanStatus(scanId, member.userId)).resolves.toMatchObject({
status: expect.any(String),
@ -40,25 +85,77 @@ describe("onboarding scan", () => {
});
it("never reads the host without a configured scan home", async () => {
const { scanId } = await startOnboardingScan({ actor: owner }, {});
const { prisma } = fakeDatabase();
const { scanId } = await startOnboardingScan({ actor: owner }, prisma);
const status = await completed(scanId, owner.userId);
expect(status.status).toBe("complete");
expect(status.result).toMatchObject({ localRepos: [], fileSummary: { totalFiles: 0 } });
});
it("lets only the deployment owner discover the configured scan home", async () => {
// The shared test setup pins the emulated runtime, which also keeps the scan simulated.
vi.stubEnv("AGENT_RUNTIME", "");
vi.stubEnv("SANDBOX_PROVIDER", "");
const scanHome = await scanHomeWithOneFile();
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app");
const { prisma } = fakeDatabase();
const asMember = await startOnboardingScan({ actor: member, scanHome }, {});
const asMember = await startOnboardingScan({ actor: member, scanHome }, prisma);
const memberStatus = await completed(asMember.scanId, member.userId);
expect(memberStatus.result).toMatchObject({ fileSummary: { totalFiles: 0 } });
expect(createSpaceForMember).not.toHaveBeenCalled();
const asOwner = await startOnboardingScan({ actor: owner, scanHome }, {});
const asOwner = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const ownerStatus = await completed(asOwner.scanId, owner.userId);
expect(ownerStatus.status).toBe("complete");
expect(ownerStatus.result).toMatchObject({ fileSummary: { totalFiles: 1 } });
});
it("links each repository of the deployment owner to one project of its own", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app", ".config/nvim");
const app = await realpath(join(scanHome, "Projects", "app"));
const { prisma, spaces } = fakeDatabase();
const first = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const firstStatus = await completed(first.scanId, owner.userId);
expect(firstStatus.status).toBe("complete");
expect(firstStatus.result).toMatchObject({
localRepos: [{ name: "app" }],
createdProjects: [{ spaceId: "space-1", name: "app", path: app }],
});
expect(createSpaceForMember).toHaveBeenCalledWith(prisma, {
currentSpaceId: owner.spaceId,
userId: owner.userId,
name: "app",
});
expect(spaces).toEqual([{ id: "space-1", name: "app", workspacePath: app }]);
// Scanning again finds the linked project instead of creating a second one.
const again = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const againStatus = await completed(again.scanId, owner.userId);
expect(againStatus.result).toMatchObject({ createdProjects: [] });
expect(createSpaceForMember).toHaveBeenCalledTimes(1);
});
it("joins a running scan instead of linking its repositories twice", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/app");
const { prisma, spaces } = fakeDatabase();
const first = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const second = await startOnboardingScan({ actor: owner, scanHome }, prisma);
expect(second.scanId).toBe(first.scanId);
await completed(first.scanId, owner.userId);
expect(spaces).toHaveLength(1);
});
it("stops creating projects at the space limit and still completes", async () => {
allowHostScan();
const scanHome = await scanHomeWithRepos("Projects/a", "Projects/b", "Projects/c");
const { prisma } = fakeDatabase({ spaceLimit: 1 });
const { scanId } = await startOnboardingScan({ actor: owner, scanHome }, prisma);
const status = await completed(scanId, owner.userId);
expect(status.status).toBe("complete");
expect(status.result).toMatchObject({ createdProjects: [{ spaceId: "space-1" }] });
expect(createSpaceForMember).toHaveBeenCalledTimes(2);
});
});

View File

@ -1,9 +1,16 @@
// Onboarding scan — real filesystem scan, Git analysis, and profile compilation.
// Runs synchronously on the host machine (no sandbox needed).
import { spawn } from "node:child_process";
import { access, readdir, stat } from "node:fs/promises";
import { dirname, join } from "node:path";
import { access, readdir, realpath, stat } from "node:fs/promises";
import { basename, dirname, join } from "node:path";
import type { Actor } from "@rakazo/contracts";
import type { PrismaClient } from "@rakazo/db";
import {
createSpaceForMember,
InvalidSpaceNameError,
SpaceLimitError,
wireProjectWorkspace,
} from "@rakazo/db";
// ---------------------------------------------------------------------------
// Types
@ -54,10 +61,9 @@ interface ProfileSummary {
}
interface CreatedProject {
id: string;
name: string;
spaceId: string;
description: string;
name: string;
path: string;
}
interface ScanRecord {
@ -115,8 +121,10 @@ async function runSimulatedScan(record: ScanRecord) {
record.completedAt = new Date();
}
type ScanActor = Pick<Actor, "userId" | "spaceId" | "isDeploymentOwner">;
export interface OnboardingScanRequest {
actor: Pick<Actor, "userId" | "isDeploymentOwner">;
actor: ScanActor;
botId?: string;
/** Home-style directory to discover. Unset keeps the scan off the host filesystem. */
scanHome?: string;
@ -124,8 +132,12 @@ export interface OnboardingScanRequest {
export async function startOnboardingScan(
{ actor, botId, scanHome }: OnboardingScanRequest,
prisma: any,
prisma: PrismaClient,
): Promise<{ scanId: string }> {
// A repeated start (a remount, a retry) joins the running scan, so no repository is linked twice.
for (const running of scanRegistry.values()) {
if (running.ownerUserId === actor.userId && !running.completedAt) return { scanId: running.id };
}
const scanId = `scan_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
const record: ScanRecord = {
id: scanId,
@ -154,7 +166,7 @@ export async function startOnboardingScan(
await runSimulatedScan(record);
return;
}
const result = await executeFullScan(scanHome, botId ?? "", prisma);
const result = await executeFullScan(scanHome, actor, prisma);
record.status = "complete";
record.progress = 100;
record.message = "Scan complete — profile ready";
@ -189,7 +201,11 @@ export async function getScanStatus(
// Full scan pipeline
// ---------------------------------------------------------------------------
async function executeFullScan(homeDir: string, _botId: string, prisma: any): Promise<ScanResult> {
async function executeFullScan(
homeDir: string,
actor: ScanActor,
prisma: PrismaClient,
): Promise<ScanResult> {
// Phase 1: Local file scan
const [fileSummary, projectFolders] = await scanFiles(homeDir);
@ -200,8 +216,8 @@ async function executeFullScan(homeDir: string, _botId: string, prisma: any): Pr
// Phase 4: Profile compilation from findings
const profileSummary = compileProfile(localRepos, fileSummary, projectFolders);
// Phase 6: Project folder creation
const createdProjects = await createProjectFolders(localRepos, prisma);
// Phase 6: Projects for discovered repositories
const createdProjects = await createProjects(localRepos, actor, prisma);
// Phase 7: Sub-agent role creation
const subAgentRoles = generateSubAgentRoles(profileSummary, localRepos);
@ -413,9 +429,9 @@ async function findGitDirectories(dir: string, depth: number = 0): Promise<strin
if (entry.name === ".git") {
results.push(dirname(fullPath));
} else if (
// Dot folders hold tools and caches (shell and editor plugins), not the person's projects.
!entry.name.startsWith(".") &&
entry.name !== "node_modules" &&
entry.name !== ".git" &&
entry.name !== ".venv" &&
entry.name !== "dist"
) {
results.push(...(await findGitDirectories(fullPath, depth + 1)));
@ -622,38 +638,43 @@ function generateBriefing(
}
// ---------------------------------------------------------------------------
// Phase 5: Project folder creation
// Phase 5: Projects for discovered repositories
// ---------------------------------------------------------------------------
async function createProjectFolders(repos: RepoInfo[], prisma: any): Promise<CreatedProject[]> {
/** Give each discovered repository a project linked to its folder, unless one already links it. */
async function createProjects(
repos: RepoInfo[],
actor: ScanActor,
prisma: PrismaClient,
): Promise<CreatedProject[]> {
const linked = await prisma.space.findMany({
where: { workspacePath: { not: null }, memberships: { some: { userId: actor.userId } } },
select: { workspacePath: true },
});
const linkedFolders = new Set(linked.map((space) => space.workspacePath));
const created: CreatedProject[] = [];
for (const repo of repos) {
// Compare and link the real folder, so the next scan recognizes the projects of this one.
const folder = await realpath(repo.path).catch(() => null);
if (!folder || linkedFolders.has(folder)) continue;
let space: { id: string; name: string };
try {
// Check if space already exists
const existing = await prisma.space.findFirst({ where: { title: repo.name } });
if (existing) {
created.push({
id: existing.id,
name: repo.name,
spaceId: existing.id,
description: repo.remote || repo.path,
});
continue;
}
// Create new space
const space = await prisma.space.create({
data: { title: repo.name, path: repo.path },
space = await createSpaceForMember(prisma, {
currentSpaceId: actor.spaceId,
userId: actor.userId,
name: basename(folder),
});
created.push({
id: space.id,
name: repo.name,
spaceId: space.id,
description: repo.remote || repo.path,
});
} catch {
// Space might already exist or DB unavailable — skip
} catch (error) {
if (error instanceof SpaceLimitError) break;
if (error instanceof InvalidSpaceNameError) continue;
throw error;
}
const workspace = await wireProjectWorkspace(prisma, {
spaceId: space.id,
workspacePath: folder,
});
created.push({ spaceId: space.id, name: space.name, path: workspace.path });
}
return created;

View File

@ -166,6 +166,7 @@ import {
renameMainConversation,
} from "./main-conversation.js";
import { buildMcpUpdateMaterial } from "./mcp-material.js";
import { createMediaRouter } from "./media.js";
import {
disconnectMemoryProvider,
getMemoryProviderConfig,
@ -526,6 +527,7 @@ export function createRouter(deps: RouterDeps) {
});
return os.router({
media: createMediaRouter({ prisma: deps.prisma, dataDir: deps.dataDir }),
aiConsent: {
status: authed.aiConsent.status.handler(({ context, input }) =>
aiConsentStatus(deps, context.actor, input),
@ -569,6 +571,10 @@ export function createRouter(deps: RouterDeps) {
spaceNavigationDto(deps, context.actor, repos, groupRepos),
),
create: authed.spaces.create.handler(async ({ context, input }) => {
// A linked folder lives on the host, and the host is the deployment owner's machine.
if (input.workspacePath && !context.actor.isDeploymentOwner) {
throw new ORPCError("FORBIDDEN");
}
let space: { id: string; name: string };
try {
space = await createSpaceForMember(deps.prisma, {
@ -2230,10 +2236,7 @@ export function createRouter(deps: RouterDeps) {
}
}
if (isProjectWorkspacePath(storedPath)) {
const workspaceRoot = await requireProjectWorkspaceRoot(
deps.prisma,
context.actor.spaceId,
);
const workspaceRoot = await requireProjectWorkspaceRoot(deps.prisma, context.actor);
const entries = await listProjectWorkspaceFiles(workspaceRoot, storedPath);
return entries.map((entry) => ({
...entry,
@ -2281,10 +2284,7 @@ export function createRouter(deps: RouterDeps) {
}
}
if (isProjectWorkspacePath(storedPath)) {
const workspaceRoot = await requireProjectWorkspaceRoot(
deps.prisma,
context.actor.spaceId,
);
const workspaceRoot = await requireProjectWorkspaceRoot(deps.prisma, context.actor);
const bytes = await readProjectWorkspaceFile(workspaceRoot, storedPath, {
maxBytes: MAX_COMPUTER_TEXT_FILE_BYTES,
});
@ -4894,10 +4894,13 @@ function mapUpdaterError(error: unknown): never {
async function requireProjectWorkspaceRoot(
prisma: Pick<PrismaClient, "space">,
spaceId: string,
actor: Actor,
): Promise<string> {
// Only the deployment owner may link a host folder, so only they may browse one. This also
// closes folders that other members linked before linking was limited to the owner.
if (!actor.isDeploymentOwner) throw new ORPCError("FORBIDDEN");
const space = await prisma.space.findUnique({
where: { id: spaceId },
where: { id: actor.spaceId },
select: { workspacePath: true },
});
if (!space?.workspacePath) {
@ -5009,7 +5012,8 @@ async function spaceNavigationDto(
memberships.length > 1 &&
!spacesWithContent.has(membership.spaceId) &&
(membership.space.deletingAt === null || membership.space.deletingAt < staleClaimBefore),
workspace: projectWorkspaceDto(membership.space),
// A linked folder is on the owner's host; other members see neither its path nor remote.
workspace: actor.isDeploymentOwner ? projectWorkspaceDto(membership.space) : null,
bots: spaceBots.map((bot) => ({
id: bot.id,
spaceId: bot.spaceId,

View File

@ -0,0 +1,2 @@
.build/
.swiftpm/

View File

@ -0,0 +1,18 @@
// swift-tools-version:5.9
import PackageDescription
// PhotosHelperCore holds everything that does not touch PhotoKit (argument parsing,
// cursors, JSON envelopes, the export path guard, hashing) so XCTest runs without
// Photos access. The executable target is the thin PhotoKit layer.
let package = Package(
name: "photos-helper",
platforms: [.macOS(.v13)],
products: [
.executable(name: "photos-helper", targets: ["photos-helper"])
],
targets: [
.target(name: "PhotosHelperCore"),
.executableTarget(name: "photos-helper", dependencies: ["PhotosHelperCore"]),
.testTarget(name: "PhotosHelperCoreTests", dependencies: ["PhotosHelperCore"]),
]
)

View File

@ -0,0 +1,142 @@
import Foundation
public enum ExportVariant: String, Equatable {
/// The untouched original resource, downloaded from iCloud when needed.
case original
/// A smaller rendition of the current version (720p video, 1280 px JPEG).
case proxy
}
public enum Command: Equatable {
/// `prompt: false` only reads the current status and never shows the system dialog.
case authorize(prompt: Bool)
case list(cursor: ListCursor?, limit: Int, since: Date?)
case asset(localIdentifier: String)
case export(localIdentifier: String, destination: String, variant: ExportVariant)
case changes(token: String)
}
public let usageText = """
usage: photos-helper authorize [--no-prompt]
photos-helper list [--cursor <opaque>] [--limit <1-1000>] [--since <iso8601>]
photos-helper asset <localIdentifier>
photos-helper export <localIdentifier> --dest <dir> [--original|--proxy]
photos-helper changes --token <opaque>
"""
/// Parses argv without the program name. Unknown flags, repeated flags and missing
/// values are usage errors so a typo never silently changes behaviour.
public func parseArguments(_ arguments: [String]) throws -> Command {
guard let name = arguments.first else { throw HelperError(.usage, usageText) }
var options = try OptionReader(Array(arguments.dropFirst()))
switch name {
case "authorize":
let noPrompt = options.flag("--no-prompt")
try options.finish(positionals: 0)
return .authorize(prompt: !noPrompt)
case "list":
let cursor = try options.value("--cursor").map(ListCursor.decode)
let limit = try options.value("--limit").map(parseLimit) ?? Limits.defaultPage
let since = try options.value("--since").map { value -> Date in
guard let date = parseTimestamp(value) else {
throw HelperError(.usage, "--since must be an ISO 8601 timestamp.")
}
return date
}
try options.finish(positionals: 0)
return .list(cursor: cursor, limit: limit, since: since)
case "asset":
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .asset(localIdentifier: identifier)
case "export":
guard let destination = options.value("--dest") else {
throw HelperError(.usage, "export needs --dest <dir>.")
}
let original = options.flag("--original")
let proxy = options.flag("--proxy")
if original && proxy { throw HelperError(.usage, "Choose --original or --proxy, not both.") }
try options.finish(positionals: 1)
let identifier = try validIdentifier(options.positionals[0])
return .export(
localIdentifier: identifier,
destination: destination,
variant: proxy ? .proxy : .original)
case "changes":
guard let token = options.value("--token"), !token.isEmpty else {
throw HelperError(.usage, "changes needs --token <opaque>.")
}
try options.finish(positionals: 0)
return .changes(token: token)
default:
throw HelperError(.usage, usageText)
}
}
func parseLimit(_ value: String) throws -> Int {
guard let limit = Int(value), (1...Limits.maxPage).contains(limit) else {
throw HelperError(.usage, "--limit must be between 1 and \(Limits.maxPage).")
}
return limit
}
func validIdentifier(_ value: String) throws -> String {
guard !value.isEmpty, value.utf16.count <= Limits.externalId, !value.hasPrefix("-"),
!value.unicodeScalars.contains(where: { CharacterSet.controlCharacters.contains($0) })
else { throw HelperError(.usage, "Invalid local identifier.") }
return value
}
private struct OptionReader {
private var values: [String: String] = [:]
private var flags: Set<String> = []
private var consumed: Set<String> = []
private(set) var positionals: [String] = []
private static let valueOptions: Set<String> = ["--cursor", "--limit", "--since", "--dest", "--token"]
private static let flagOptions: Set<String> = ["--no-prompt", "--original", "--proxy"]
init(_ arguments: [String]) throws {
var index = 0
while index < arguments.count {
let argument = arguments[index]
if Self.valueOptions.contains(argument) {
guard index + 1 < arguments.count else {
throw HelperError(.usage, "\(argument) needs a value.")
}
guard values[argument] == nil else { throw HelperError(.usage, "\(argument) given twice.") }
values[argument] = arguments[index + 1]
index += 2
} else if Self.flagOptions.contains(argument) {
guard !flags.contains(argument) else { throw HelperError(.usage, "\(argument) given twice.") }
flags.insert(argument)
index += 1
} else if argument.hasPrefix("--") {
throw HelperError(.usage, "Unknown option \(argument).")
} else {
positionals.append(argument)
index += 1
}
}
}
mutating func value(_ name: String) -> String? {
consumed.insert(name)
return values[name]
}
mutating func flag(_ name: String) -> Bool {
consumed.insert(name)
return flags.contains(name)
}
/// Rejects options that belong to another command and surplus positionals.
func finish(positionals expected: Int) throws {
for name in Set(values.keys).union(flags) where !consumed.contains(name) {
throw HelperError(.usage, "\(name) is not valid here.")
}
guard positionals.count == expected else {
throw HelperError(.usage, expected == 0 ? "Unexpected argument." : "Expected \(expected) argument.")
}
}
}

View File

@ -0,0 +1,37 @@
import Foundation
/// A failure the desktop app maps by `code`. Messages never contain library content
/// beyond the identifier the caller passed in.
public struct HelperError: Error, Equatable {
public enum Code: String, Codable, CaseIterable {
case usage
case notAuthorized = "not_authorized"
case notFound = "not_found"
case pathRejected = "path_rejected"
case destinationExists = "destination_exists"
case network
case exportFailed = "export_failed"
case `internal`
}
public let code: Code
public let message: String
public init(_ code: Code, _ message: String) {
self.code = code
self.message = message
}
/// sysexits(3) style codes so a shell user can tell failure classes apart.
public var exitCode: Int32 {
switch code {
case .usage: return 64
case .notFound: return 66
case .pathRejected, .destinationExists: return 73
case .exportFailed: return 74
case .network: return 75
case .notAuthorized: return 77
case .internal: return 70
}
}
}

View File

@ -0,0 +1,249 @@
import Foundation
/// Mirrors `ClientMediaItemSchema` in packages/contracts/src/media.ts. Every key is
/// always present (nullable fields encode as `null`) because the schema has no optional keys.
public struct ClientMediaItem: Encodable, Equatable {
public enum Kind: String, Encodable {
case video
case photo
case livePhoto = "live_photo"
}
public var externalId: String
public var kind: Kind
public var capturedAt: String?
public var durationMs: Int?
public var width: Int?
public var height: Int?
public var fps: Double?
public var gpsLat: Double?
public var gpsLon: Double?
public var deviceModel: String?
public var albumNames: [String]
public var isFavorite: Bool
public var fileName: String?
public init(
externalId: String,
kind: Kind,
capturedAt: Date?,
durationSeconds: Double?,
width: Int?,
height: Int?,
latitude: Double?,
longitude: Double?,
albumNames: [String],
isFavorite: Bool,
fileName: String?
) {
self.externalId = externalId
self.kind = kind
self.capturedAt = capturedAt.map(formatTimestamp)
if let durationSeconds, durationSeconds.isFinite, durationSeconds > 0 {
durationMs = Int((durationSeconds * 1000).rounded())
} else {
durationMs = nil
}
self.width = width.flatMap { $0 > 0 ? $0 : nil }
self.height = height.flatMap { $0 > 0 ? $0 : nil }
// PhotoKit exposes neither frame rate nor camera model without loading the
// file; the server reads both with ffprobe after upload.
fps = nil
deviceModel = nil
let coordinate = roundedCoordinate(latitude: latitude, longitude: longitude)
gpsLat = coordinate?.latitude
gpsLon = coordinate?.longitude
self.albumNames = Array(
albumNames.map { clip($0, maxCharacters: Limits.albumName) }
.filter { !$0.isEmpty }
.prefix(Limits.albumCount))
self.isFavorite = isFavorite
self.fileName = fileName.map { clip($0, maxCharacters: Limits.fileName) }
}
enum CodingKeys: String, CodingKey {
case externalId, kind, capturedAt, durationMs, width, height, fps, gpsLat, gpsLon
case deviceModel, albumNames, isFavorite, fileName
}
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(externalId, forKey: .externalId)
try container.encode(kind, forKey: .kind)
// `encode` on an Optional writes `null`, unlike the synthesized `encodeIfPresent`.
try container.encode(capturedAt, forKey: .capturedAt)
try container.encode(durationMs, forKey: .durationMs)
try container.encode(width, forKey: .width)
try container.encode(height, forKey: .height)
try container.encode(fps, forKey: .fps)
try container.encode(gpsLat, forKey: .gpsLat)
try container.encode(gpsLon, forKey: .gpsLon)
try container.encode(deviceModel, forKey: .deviceModel)
try container.encode(albumNames, forKey: .albumNames)
try container.encode(isFavorite, forKey: .isFavorite)
try container.encode(fileName, forKey: .fileName)
}
}
/// Bounds from the contract schema, applied here so one odd asset cannot make a whole
/// page fail validation in the app.
public enum Limits {
public static let externalId = 200
public static let albumName = 200
public static let albumCount = 50
public static let fileName = 255
public static let defaultPage = 500
public static let maxPage = 1000
/// Above this many changed assets a full relisting is cheaper than one huge document.
public static let maxChanges = 5000
}
/// Clips to `maxCharacters` UTF-16 units (what zod's `.max()` counts) without splitting
/// a character.
public func clip(_ value: String, maxCharacters: Int) -> String {
if value.utf16.count <= maxCharacters { return value }
var result = ""
var units = 0
for character in value {
units += character.utf16.count
if units > maxCharacters { break }
result.append(character)
}
return result
}
/// Two decimals is about 1 km, matching `roundCoordinate` in @rakazo/media. Precise
/// locations never leave the device.
public func roundedCoordinate(latitude: Double?, longitude: Double?) -> (
latitude: Double, longitude: Double
)? {
guard let latitude, let longitude, latitude.isFinite, longitude.isFinite,
(-90...90).contains(latitude), (-180...180).contains(longitude),
!(latitude == 0 && longitude == 0)
else { return nil }
return ((latitude * 100).rounded() / 100, (longitude * 100).rounded() / 100)
}
/// RFC 3339 in UTC with milliseconds, accepted by zod's `datetime({ offset: true })`.
public func formatTimestamp(_ date: Date) -> String {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
formatter.timeZone = TimeZone(identifier: "UTC")
return formatter.string(from: date)
}
public func parseTimestamp(_ value: String) -> Date? {
let formatter = ISO8601DateFormatter()
formatter.formatOptions = [.withInternetDateTime, .withFractionalSeconds]
if let date = formatter.date(from: value) { return date }
formatter.formatOptions = [.withInternetDateTime]
return formatter.date(from: value)
}
public enum AuthorizationState: String, Encodable {
case authorized, limited, denied, restricted, notDetermined
}
public struct AuthorizationOutput: Encodable, Equatable {
public let status: AuthorizationState
public init(status: AuthorizationState) { self.status = status }
}
public struct ListOutput: Encodable {
public let items: [ClientMediaItem]
public let nextCursor: String?
/// Library change token captured before enumeration; `changes --token` resumes from it.
public let changeToken: String?
public init(items: [ClientMediaItem], nextCursor: String?, changeToken: String?) {
self.items = items
self.nextCursor = nextCursor
self.changeToken = changeToken
}
enum CodingKeys: String, CodingKey { case items, nextCursor, changeToken }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(items, forKey: .items)
try container.encode(nextCursor, forKey: .nextCursor)
try container.encode(changeToken, forKey: .changeToken)
}
}
public struct AssetResourceInfo: Encodable, Equatable {
public let type: String
public let fileName: String
public let uti: String
public init(type: String, fileName: String, uti: String) {
self.type = type
self.fileName = fileName
self.uti = uti
}
}
public struct AssetOutput: Encodable {
public let item: ClientMediaItem
public let modifiedAt: String?
public let hidden: Bool
public let resources: [AssetResourceInfo]
public init(item: ClientMediaItem, modifiedAt: Date?, hidden: Bool, resources: [AssetResourceInfo]) {
self.item = item
self.modifiedAt = modifiedAt.map(formatTimestamp)
self.hidden = hidden
self.resources = resources
}
enum CodingKeys: String, CodingKey { case item, modifiedAt, hidden, resources }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(item, forKey: .item)
try container.encode(modifiedAt, forKey: .modifiedAt)
try container.encode(hidden, forKey: .hidden)
try container.encode(resources, forKey: .resources)
}
}
public struct ChangesOutput: Encodable {
/// Token to store for the next call; null only when a full relisting is required.
public let token: String?
public let changed: [ClientMediaItem]
public let deleted: [String]
public let fullResyncRequired: Bool
public init(token: String?, changed: [ClientMediaItem], deleted: [String], fullResyncRequired: Bool) {
self.token = token
self.changed = changed
self.deleted = deleted
self.fullResyncRequired = fullResyncRequired
}
public static func fullResync() -> ChangesOutput {
ChangesOutput(token: nil, changed: [], deleted: [], fullResyncRequired: true)
}
enum CodingKeys: String, CodingKey { case token, changed, deleted, fullResyncRequired }
public func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(token, forKey: .token)
try container.encode(changed, forKey: .changed)
try container.encode(deleted, forKey: .deleted)
try container.encode(fullResyncRequired, forKey: .fullResyncRequired)
}
}
public struct ExportOutput: Encodable, Equatable {
public let path: String
public let fileName: String
public let sizeBytes: Int64
public let sha256: String
public init(path: String, fileName: String, sizeBytes: Int64, sha256: String) {
self.path = path
self.fileName = fileName
self.sizeBytes = sizeBytes
self.sha256 = sha256
}
}

View File

@ -0,0 +1,56 @@
import CryptoKit
import Foundation
/// stdout carries exactly one JSON document per successful command; stderr carries
/// JSON lines (`{"progress":…}` while exporting, `{"error":…}` on failure).
public func encodeJSON<T: Encodable>(_ value: T) throws -> String {
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes]
encoder.nonConformingFloatEncodingStrategy = .throw
let data = try encoder.encode(value)
guard let text = String(data: data, encoding: .utf8) else {
throw HelperError(.internal, "Output was not UTF-8.")
}
return text
}
private struct ErrorEnvelope: Encodable {
struct Body: Encodable {
let code: String
let message: String
}
let error: Body
}
public func errorLine(_ error: HelperError) -> String {
let envelope = ErrorEnvelope(error: .init(code: error.code.rawValue, message: error.message))
return (try? encodeJSON(envelope))
?? #"{"error":{"code":"internal","message":"Could not encode the error."}}"#
}
private struct ProgressLine: Encodable {
let progress: Double
}
/// Progress clamped to 0…1 and rounded to three decimals so the line stays short.
public func progressLine(_ fraction: Double) -> String {
let value = fraction.isFinite ? min(1, max(0, fraction)) : 0
return (try? encodeJSON(ProgressLine(progress: (value * 1000).rounded() / 1000)))
?? #"{"progress":0}"#
}
/// Streams the file through SHA-256 in 1 MiB reads so large videos never load fully.
public func sha256Hex(of url: URL) throws -> (hex: String, sizeBytes: Int64) {
let handle = try FileHandle(forReadingFrom: url)
defer { try? handle.close() }
var hasher = SHA256()
var size: Int64 = 0
while true {
let chunk = try handle.read(upToCount: 1 << 20) ?? Data()
if chunk.isEmpty { break }
hasher.update(data: chunk)
size += Int64(chunk.count)
}
let hex = hasher.finalize().map { String(format: "%02x", $0) }.joined()
return (hex, size)
}

View File

@ -0,0 +1,131 @@
import Foundation
/// Listing order: undated assets first, then creation date ascending, ties broken by
/// local identifier. PhotoKit can only sort by date, so ties are ordered here.
public struct SortKey: Comparable, Hashable {
/// Milliseconds since 1970; nil for assets without a creation date.
public let creationMillis: Int64?
public let localIdentifier: String
public init(creationMillis: Int64?, localIdentifier: String) {
self.creationMillis = creationMillis
self.localIdentifier = localIdentifier
}
public init(creationDate: Date?, localIdentifier: String) {
self.init(
creationMillis: creationDate.map { Int64(($0.timeIntervalSince1970 * 1000).rounded(.down)) },
localIdentifier: localIdentifier)
}
public static func < (lhs: SortKey, rhs: SortKey) -> Bool {
if lhs.creationMillis != rhs.creationMillis {
switch (lhs.creationMillis, rhs.creationMillis) {
case (nil, _): return true
case (_, nil): return false
case let (left?, right?): return left < right
}
}
return lhs.localIdentifier < rhs.localIdentifier
}
}
/// Opaque to the app: base64url of `{"v":1,"d":<millis|null>,"id":"<localIdentifier>"}`.
/// The cursor names the last item returned, so listing resumes after it even when
/// assets were added or removed in between.
public struct ListCursor: Equatable {
public let key: SortKey
public init(key: SortKey) { self.key = key }
private struct Payload: Codable {
let v: Int
let d: Int64?
let id: String
enum CodingKeys: String, CodingKey { case v, d, id }
func encode(to encoder: Encoder) throws {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(v, forKey: .v)
try container.encode(d, forKey: .d)
try container.encode(id, forKey: .id)
}
}
public func encoded() -> String {
let payload = Payload(v: 1, d: key.creationMillis, id: key.localIdentifier)
let encoder = JSONEncoder()
encoder.outputFormatting = [.sortedKeys]
// Encoding a struct of plain values cannot fail.
let data = (try? encoder.encode(payload)) ?? Data()
return base64URLEncode(data)
}
public static func decode(_ value: String) throws -> ListCursor {
guard value.count <= 2048, let data = base64URLDecode(value),
let payload = try? JSONDecoder().decode(Payload.self, from: data),
payload.v == 1, !payload.id.isEmpty
else { throw HelperError(.usage, "Invalid cursor.") }
return ListCursor(key: SortKey(creationMillis: payload.d, localIdentifier: payload.id))
}
}
public func base64URLEncode(_ data: Data) -> String {
data.base64EncodedString()
.replacingOccurrences(of: "+", with: "-")
.replacingOccurrences(of: "/", with: "_")
.replacingOccurrences(of: "=", with: "")
}
public func base64URLDecode(_ value: String) -> Data? {
guard value.allSatisfy({ $0.isASCII && ($0.isLetter || $0.isNumber || $0 == "-" || $0 == "_") })
else { return nil }
var base64 = value.replacingOccurrences(of: "-", with: "+").replacingOccurrences(of: "_", with: "/")
while base64.count % 4 != 0 { base64.append("=") }
return Data(base64Encoded: base64)
}
public struct Page {
/// Positions in the source sequence, in listing order.
public let indices: [Int]
public let hasMore: Bool
}
/// Selects the next page from a sequence sorted by creation date only (ties in any
/// order). Binary search finds the first candidate date; a whole same-date group is
/// collected before sorting so tie order never depends on PhotoKit.
public func nextPage(count: Int, key: (Int) -> SortKey, after cursor: ListCursor?, limit: Int)
-> Page
{
precondition(limit > 0)
var start = 0
if let cursor, let cursorMillis = cursor.key.creationMillis {
var low = 0
var high = count
while low < high {
let mid = (low + high) / 2
let millis = key(mid).creationMillis
if millis == nil || millis! < cursorMillis { low = mid + 1 } else { high = mid }
}
start = low
}
var candidates: [(index: Int, key: SortKey)] = []
var stoppedEarly = false
var index = start
while index < count {
let current = key(index)
if candidates.count >= limit, let last = candidates.last, current.creationMillis != last.key.creationMillis {
// Dates ascend, so everything from here on sorts after the collected group.
stoppedEarly = true
break
}
if cursor == nil || cursor!.key < current {
candidates.append((index, current))
}
index += 1
}
candidates.sort { $0.key < $1.key }
let hasMore = stoppedEarly || candidates.count > limit
return Page(indices: candidates.prefix(limit).map(\.index), hasMore: hasMore)
}

View File

@ -0,0 +1,72 @@
import Foundation
/// Where an export is written: a final path and a sibling temporary path inside the
/// destination directory.
public struct ExportDestination: Equatable {
public let directory: URL
public let finalURL: URL
public let temporaryURL: URL
public let fileName: String
}
/// Resolves the export target. The directory must be absolute, free of `..`, existing
/// and a directory after resolving symlinks (macOS temp dirs live behind /var). The file
/// name comes from Photos, so it is reduced to one safe component and the joined path is
/// checked to stay a direct child of the resolved directory. Existing files are never
/// overwritten.
public func resolveExportDestination(
directory: String,
fileName rawFileName: String,
temporarySuffix: String = UUID().uuidString,
fileManager: FileManager = .default
) throws -> ExportDestination {
guard directory.hasPrefix("/"), !directory.contains("\0") else {
throw HelperError(.pathRejected, "--dest must be an absolute directory path.")
}
let components = directory.split(separator: "/", omittingEmptySubsequences: true)
guard !components.contains(where: { $0 == ".." || $0 == "." }) else {
throw HelperError(.pathRejected, "--dest must not contain . or .. segments.")
}
let resolved = URL(fileURLWithPath: directory, isDirectory: true).resolvingSymlinksInPath()
.standardizedFileURL
var isDirectory: ObjCBool = false
guard fileManager.fileExists(atPath: resolved.path, isDirectory: &isDirectory), isDirectory.boolValue
else { throw HelperError(.pathRejected, "--dest must be an existing directory.") }
let fileName = sanitizeFileName(rawFileName)
let finalURL = resolved.appendingPathComponent(fileName, isDirectory: false).standardizedFileURL
let temporaryURL = resolved.appendingPathComponent(
".\(fileName).\(temporarySuffix).partial", isDirectory: false
).standardizedFileURL
for candidate in [finalURL, temporaryURL]
where candidate.deletingLastPathComponent().path != resolved.path {
throw HelperError(.pathRejected, "Export path left the destination directory.")
}
// lstat semantics: a dangling symlink at the final path also counts as existing.
if (try? fileManager.attributesOfItem(atPath: finalURL.path)) != nil {
throw HelperError(.destinationExists, "The destination file already exists.")
}
return ExportDestination(
directory: resolved, finalURL: finalURL, temporaryURL: temporaryURL, fileName: fileName)
}
/// One path component with no separators, control characters or leading dots, at most
/// 200 UTF-8 bytes with the extension kept.
public func sanitizeFileName(_ raw: String) -> String {
var cleaned = String(
raw.unicodeScalars.map { scalar -> Character in
if scalar == "/" || scalar == "\\" || scalar == ":" { return "_" }
if CharacterSet.controlCharacters.contains(scalar) { return "_" }
return Character(scalar)
})
while cleaned.hasPrefix(".") || cleaned.hasPrefix(" ") { cleaned.removeFirst() }
cleaned = cleaned.trimmingCharacters(in: .whitespaces)
if cleaned.isEmpty { return "asset" }
let maxBytes = 200
if cleaned.utf8.count <= maxBytes { return cleaned }
let ext = (cleaned as NSString).pathExtension
let suffix = ext.isEmpty || ext.utf8.count > 16 ? "" : ".\(ext)"
var stem = suffix.isEmpty ? cleaned : String(cleaned.dropLast(suffix.count))
while stem.utf8.count + suffix.utf8.count > maxBytes { stem.removeLast() }
return stem + suffix
}

View File

@ -0,0 +1,148 @@
import AVFoundation
import AppKit
import Foundation
import Photos
import PhotosHelperCore
/// Writes one asset into the destination directory: first to a hidden temporary file,
/// then renamed into place after hashing, so a cancelled export never leaves a file
/// that looks complete.
struct Exporter {
func export(
localIdentifier: String,
directory: String,
variant: ExportVariant,
progress: @escaping (Double) -> Void
) async throws -> ExportOutput {
guard
let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil)
.firstObject
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
guard let resource = originalResource(for: asset, variant: variant) else {
throw HelperError(.notFound, "The asset has no exportable resource.")
}
var fileName = resource.originalFilename
if variant == .proxy {
let stem = (fileName as NSString).deletingPathExtension
fileName = stem + (asset.mediaType == .video ? ".mp4" : ".jpg")
}
let destination = try resolveExportDestination(directory: directory, fileName: fileName)
let fileManager = FileManager.default
do {
switch (variant, asset.mediaType) {
case (.original, _):
try await writeResource(resource, to: destination.temporaryURL, progress: progress)
case (.proxy, .video):
try await writeVideoProxy(asset, to: destination.temporaryURL, progress: progress)
case (.proxy, _):
try await writeImageProxy(asset, to: destination.temporaryURL, progress: progress)
}
let digest = try sha256Hex(of: destination.temporaryURL)
// moveItem refuses an existing destination, so nothing is ever overwritten.
try fileManager.moveItem(at: destination.temporaryURL, to: destination.finalURL)
progress(1)
return ExportOutput(
path: destination.finalURL.path, fileName: destination.fileName,
sizeBytes: digest.sizeBytes, sha256: digest.hex)
} catch {
try? fileManager.removeItem(at: destination.temporaryURL)
if let helperError = error as? HelperError { throw helperError }
throw mapExportError(error)
}
}
/// Original bytes through PHAssetResourceManager; downloads from iCloud when the
/// library keeps only optimized copies on this Mac.
private func writeResource(
_ resource: PHAssetResource, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHAssetResourceRequestOptions()
options.isNetworkAccessAllowed = true
options.progressHandler = { value in progress(value) }
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
PHAssetResourceManager.default().writeData(for: resource, toFile: url, options: options) {
error in
if let error { continuation.resume(throwing: error) } else { continuation.resume() }
}
}
}
/// 720p H.264 MP4 of the current version, without source metadata.
private func writeVideoProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHVideoRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.version = .current
options.progressHandler = { value, _, _, _ in progress(value * 0.5) }
let session: AVAssetExportSession = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestExportSession(
forVideo: asset, options: options, exportPreset: AVAssetExportPreset1280x720
) { session, info in
if let session {
continuation.resume(returning: session)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not prepare the video export."))
}
}
}
session.outputURL = url
session.outputFileType = .mp4
session.shouldOptimizeForNetworkUse = true
session.metadata = []
session.metadataItemFilter = .forSharing()
await session.export()
guard session.status == .completed else {
throw session.error ?? HelperError(.exportFailed, "The video export did not finish.")
}
}
/// JPEG of the current version, at most 1280 px on the long edge, without EXIF/GPS.
private func writeImageProxy(
_ asset: PHAsset, to url: URL, progress: @escaping (Double) -> Void
) async throws {
let options = PHImageRequestOptions()
options.isNetworkAccessAllowed = true
options.deliveryMode = .highQualityFormat
options.resizeMode = .exact
options.version = .current
options.isSynchronous = false
options.progressHandler = { value, _, _, _ in progress(value) }
// highQualityFormat calls the handler exactly once.
let image: NSImage = try await withCheckedThrowingContinuation { continuation in
PHImageManager.default().requestImage(
for: asset, targetSize: CGSize(width: 1280, height: 1280), contentMode: .aspectFit,
options: options
) { image, info in
if let image {
continuation.resume(returning: image)
} else {
continuation.resume(
throwing: (info?[PHImageErrorKey] as? Error)
?? HelperError(.exportFailed, "Could not render the photo."))
}
}
}
guard let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil),
let data = NSBitmapImageRep(cgImage: cgImage).representation(
using: .jpeg, properties: [.compressionFactor: 0.85])
else { throw HelperError(.exportFailed, "Could not encode the photo.") }
try data.write(to: url, options: .withoutOverwriting)
}
}
/// iCloud and network failures are retryable; everything else is an export failure.
func mapExportError(_ error: Error) -> HelperError {
if let photosError = error as? PHPhotosError,
photosError.code == .networkAccessRequired || photosError.code == .networkError
{
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
if (error as NSError).domain == NSURLErrorDomain {
return HelperError(.network, "The original could not be downloaded from iCloud.")
}
return HelperError(.exportFailed, "The export failed.")
}

View File

@ -0,0 +1,190 @@
import Foundation
import Photos
import PhotosHelperCore
/// Read-only PhotoKit queries. Nothing here writes to the library.
struct Library {
private let library = PHPhotoLibrary.shared()
/// Images and videos only; audio and unknown media types are not part of Media Studio.
/// PhotoKit sorts by date only; the store places missing dates first, which is what
/// `SortKey` assumes, and `nextPage` orders ties itself.
private func fetchOptions(since: Date?) -> PHFetchOptions {
let options = PHFetchOptions()
options.includeHiddenAssets = false
var predicates = [
NSPredicate(
format: "mediaType == %ld || mediaType == %ld",
PHAssetMediaType.image.rawValue, PHAssetMediaType.video.rawValue)
]
if let since {
predicates.append(NSPredicate(format: "modificationDate >= %@", since as NSDate))
}
options.predicate = NSCompoundPredicate(andPredicateWithSubpredicates: predicates)
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: true)]
return options
}
func list(after cursor: ListCursor?, limit: Int, since: Date?) throws -> ListOutput {
// Captured before enumeration so changes made while listing show up next time.
let changeToken = try encodeToken(library.currentChangeToken)
let result = PHAsset.fetchAssets(with: fetchOptions(since: since))
let page = nextPage(
count: result.count,
key: { index in
let asset = result.object(at: index)
return SortKey(creationDate: asset.creationDate, localIdentifier: asset.localIdentifier)
},
after: cursor,
limit: limit)
let assets = page.indices.map { result.object(at: $0) }
let items = assets.compactMap(makeItem(for:))
let nextCursor = page.hasMore
? assets.last.map {
ListCursor(key: SortKey(creationDate: $0.creationDate, localIdentifier: $0.localIdentifier))
.encoded()
}
: nil
return ListOutput(items: items, nextCursor: nextCursor, changeToken: changeToken)
}
func asset(_ localIdentifier: String) throws -> AssetOutput {
guard let asset = PHAsset.fetchAssets(withLocalIdentifiers: [localIdentifier], options: nil).firstObject,
let item = makeItem(for: asset)
else { throw HelperError(.notFound, "No photo or video with that identifier.") }
let resources = PHAssetResource.assetResources(for: asset).map {
AssetResourceInfo(
type: resourceTypeName($0.type), fileName: $0.originalFilename, uti: $0.uniformTypeIdentifier)
}
return AssetOutput(
item: item, modifiedAt: asset.modificationDate, hidden: asset.isHidden, resources: resources)
}
/// Persistent change history (macOS 13). An expired token or missing history asks the
/// caller for a full relisting instead of guessing.
func changes(since encodedToken: String) throws -> ChangesOutput {
guard let token = decodeToken(encodedToken) else { return .fullResync() }
let current = try encodeToken(library.currentChangeToken)
var changed = Set<String>()
var deleted = Set<String>()
do {
let result = try library.fetchPersistentChanges(since: token)
var detailsError: Error?
result.enumerateChanges { change, stop in
do {
let details = try change.changeDetails(for: .asset)
changed.formUnion(details.insertedLocalIdentifiers)
changed.formUnion(details.updatedLocalIdentifiers)
deleted.formUnion(details.deletedLocalIdentifiers)
if changed.count + deleted.count > Limits.maxChanges { stop.pointee = true }
} catch {
detailsError = error
stop.pointee = true
}
}
if let detailsError { throw detailsError }
} catch let error as PHPhotosError
where error.code == .persistentChangeTokenExpired
|| error.code == .persistentChangeDetailsUnavailable
{
return .fullResync()
} catch {
throw HelperError(.internal, "Could not read the Photos change history.")
}
if changed.count + deleted.count > Limits.maxChanges { return .fullResync() }
changed.subtract(deleted)
var items: [ClientMediaItem] = []
var found = Set<String>()
if !changed.isEmpty {
PHAsset.fetchAssets(withLocalIdentifiers: Array(changed), options: nil).enumerateObjects {
asset, _, _ in
found.insert(asset.localIdentifier)
if let item = makeItem(for: asset) { items.append(item) }
}
}
// Assets that vanished between the history read and the fetch count as deleted.
deleted.formUnion(changed.subtracting(found))
items.sort { $0.externalId < $1.externalId }
return ChangesOutput(
token: current, changed: items, deleted: deleted.sorted(), fullResyncRequired: false)
}
private func encodeToken(_ token: PHPersistentChangeToken) throws -> String {
guard
let data = try? NSKeyedArchiver.archivedData(withRootObject: token, requiringSecureCoding: true)
else { throw HelperError(.internal, "Could not encode the Photos change token.") }
return base64URLEncode(data)
}
private func decodeToken(_ value: String) -> PHPersistentChangeToken? {
guard let data = base64URLDecode(value) else { return nil }
return try? NSKeyedUnarchiver.unarchivedObject(ofClass: PHPersistentChangeToken.self, from: data)
}
}
/// Builds the contract item. Returns nil for media types Media Studio does not handle
/// and for identifiers longer than the contract allows.
func makeItem(for asset: PHAsset) -> ClientMediaItem? {
let kind: ClientMediaItem.Kind
switch asset.mediaType {
case .video: kind = .video
case .image: kind = asset.mediaSubtypes.contains(.photoLive) ? .livePhoto : .photo
default: return nil
}
guard asset.localIdentifier.utf16.count <= Limits.externalId else { return nil }
var albums: [String] = []
PHAssetCollection.fetchAssetCollectionsContaining(asset, with: .album, options: nil)
.enumerateObjects { collection, _, stop in
if let title = collection.localizedTitle { albums.append(title) }
if albums.count >= Limits.albumCount { stop.pointee = true }
}
let coordinate = asset.location?.coordinate
return ClientMediaItem(
externalId: asset.localIdentifier,
kind: kind,
capturedAt: asset.creationDate,
durationSeconds: kind == .video ? asset.duration : nil,
width: asset.pixelWidth,
height: asset.pixelHeight,
latitude: coordinate?.latitude,
longitude: coordinate?.longitude,
albumNames: albums,
isFavorite: asset.isFavorite,
fileName: originalResource(for: asset, variant: .original)?.originalFilename)
}
/// The resource holding the untouched original: the video track for videos, the still
/// for photos and Live Photos (the paired motion clip is not exported).
func originalResource(for asset: PHAsset, variant: ExportVariant) -> PHAssetResource? {
let resources = PHAssetResource.assetResources(for: asset)
let preferred: [PHAssetResourceType]
switch (asset.mediaType, variant) {
case (.video, .original): preferred = [.video, .fullSizeVideo]
case (.video, .proxy): preferred = [.fullSizeVideo, .video]
case (_, .original): preferred = [.photo, .fullSizePhoto]
case (_, .proxy): preferred = [.fullSizePhoto, .photo]
}
for type in preferred {
if let match = resources.first(where: { $0.type == type }) { return match }
}
return nil
}
func resourceTypeName(_ type: PHAssetResourceType) -> String {
switch type {
case .photo: return "photo"
case .video: return "video"
case .audio: return "audio"
case .alternatePhoto: return "alternate_photo"
case .fullSizePhoto: return "full_size_photo"
case .fullSizeVideo: return "full_size_video"
case .adjustmentData: return "adjustment_data"
case .adjustmentBasePhoto: return "adjustment_base_photo"
case .pairedVideo: return "paired_video"
case .fullSizePairedVideo: return "full_size_paired_video"
case .adjustmentBasePairedVideo: return "adjustment_base_paired_video"
case .adjustmentBaseVideo: return "adjustment_base_video"
@unknown default: return "other"
}
}

View File

@ -0,0 +1,103 @@
import Foundation
import Photos
import PhotosHelperCore
/// Entry point. Exactly one JSON document goes to stdout on success; failures write
/// `{"error":{code,message}}` to stderr and exit non-zero (see HelperError.exitCode).
@main
struct PhotosHelper {
static func main() async {
do {
let command = try parseArguments(Array(CommandLine.arguments.dropFirst()))
let output = try await run(command)
FileHandle.standardOutput.write(Data((output + "\n").utf8))
exit(0)
} catch let error as HelperError {
fail(error)
} catch {
fail(HelperError(.internal, "Unexpected failure."))
}
}
static func fail(_ error: HelperError) -> Never {
FileHandle.standardError.write(Data((errorLine(error) + "\n").utf8))
exit(error.exitCode)
}
static func run(_ command: Command) async throws -> String {
switch command {
case .authorize(let prompt):
let status = await authorize(prompt: prompt)
return try encodeJSON(AuthorizationOutput(status: status))
case .list(let cursor, let limit, let since):
try requireAccess()
return try encodeJSON(try Library().list(after: cursor, limit: limit, since: since))
case .asset(let localIdentifier):
try requireAccess()
return try encodeJSON(try Library().asset(localIdentifier))
case .export(let localIdentifier, let destination, let variant):
try requireAccess()
let progress = ProgressReporter()
// Local copies report no progress, so a heartbeat repeats the last value; the app
// treats a silent helper as stalled.
let heartbeat = Task {
while !Task.isCancelled {
try? await Task.sleep(nanoseconds: 15_000_000_000)
if !Task.isCancelled { progress.emit(nil) }
}
}
defer { heartbeat.cancel() }
progress.emit(0)
let result = try await Exporter().export(
localIdentifier: localIdentifier, directory: destination, variant: variant,
progress: { fraction in progress.emit(fraction) })
return try encodeJSON(result)
case .changes(let token):
try requireAccess()
return try encodeJSON(try Library().changes(since: token))
}
}
/// `.readWrite` is the lowest PhotoKit level that can read assets: `.addOnly` cannot
/// enumerate or export anything. The helper never writes to the library.
static func authorize(prompt: Bool) async -> AuthorizationState {
var status = PHPhotoLibrary.authorizationStatus(for: .readWrite)
if prompt && status == .notDetermined {
status = await PHPhotoLibrary.requestAuthorization(for: .readWrite)
}
return state(status)
}
static func state(_ status: PHAuthorizationStatus) -> AuthorizationState {
switch status {
case .authorized: return .authorized
case .limited: return .limited
case .denied: return .denied
case .restricted: return .restricted
case .notDetermined: return .notDetermined
@unknown default: return .denied
}
}
/// Serializes progress lines written from PhotoKit callback queues and the heartbeat.
final class ProgressReporter: @unchecked Sendable {
private let lock = NSLock()
private var last = 0.0
/// nil repeats the last reported value.
func emit(_ fraction: Double?) {
lock.lock()
defer { lock.unlock() }
if let fraction { last = max(last, fraction) }
FileHandle.standardError.write(Data((progressLine(last) + "\n").utf8))
}
}
/// Every command except `authorize` runs only with granted access and never prompts.
static func requireAccess() throws {
switch state(PHPhotoLibrary.authorizationStatus(for: .readWrite)) {
case .authorized, .limited: return
default: throw HelperError(.notAuthorized, "Photos access has not been granted.")
}
}
}

View File

@ -0,0 +1,73 @@
import XCTest
@testable import PhotosHelperCore
final class ArgumentsTests: XCTestCase {
func testAuthorizePromptsUnlessAskedNotTo() throws {
XCTAssertEqual(try parseArguments(["authorize"]), .authorize(prompt: true))
XCTAssertEqual(try parseArguments(["authorize", "--no-prompt"]), .authorize(prompt: false))
}
func testListDefaultsAndOptions() throws {
XCTAssertEqual(try parseArguments(["list"]), .list(cursor: nil, limit: 500, since: nil))
let cursor = ListCursor(key: SortKey(creationMillis: 1_700_000_000_000, localIdentifier: "A/L0/001"))
let parsed = try parseArguments([
"list", "--cursor", cursor.encoded(), "--limit", "25", "--since", "2024-01-02T03:04:05Z",
])
XCTAssertEqual(
parsed,
.list(cursor: cursor, limit: 25, since: Date(timeIntervalSince1970: 1_704_164_645)))
}
func testListRejectsBadValues() {
for arguments in [
["list", "--limit", "0"],
["list", "--limit", "1001"],
["list", "--limit", "ten"],
["list", "--limit"],
["list", "--since", "yesterday"],
["list", "--cursor", "not a cursor"],
["list", "--limit", "5", "--limit", "6"],
["list", "--dest", "/tmp"],
["list", "extra"],
["list", "--unknown"],
] {
XCTAssertThrowsError(try parseArguments(arguments), "\(arguments)") { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
}
}
}
func testAssetNeedsExactlyOneIdentifier() throws {
XCTAssertEqual(try parseArguments(["asset", "ABC/L0/001"]), .asset(localIdentifier: "ABC/L0/001"))
XCTAssertThrowsError(try parseArguments(["asset"]))
XCTAssertThrowsError(try parseArguments(["asset", "a", "b"]))
XCTAssertThrowsError(try parseArguments(["asset", String(repeating: "x", count: 201)]))
XCTAssertThrowsError(try parseArguments(["asset", "bad\nid"]))
}
func testExportVariants() throws {
XCTAssertEqual(
try parseArguments(["export", "ID", "--dest", "/tmp/out"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .original))
XCTAssertEqual(
try parseArguments(["export", "--dest", "/tmp/out", "ID", "--proxy"]),
.export(localIdentifier: "ID", destination: "/tmp/out", variant: .proxy))
XCTAssertThrowsError(try parseArguments(["export", "ID"]))
XCTAssertThrowsError(try parseArguments(["export", "ID", "--dest", "/tmp", "--original", "--proxy"]))
}
func testChangesNeedsToken() throws {
XCTAssertEqual(try parseArguments(["changes", "--token", "abc"]), .changes(token: "abc"))
XCTAssertThrowsError(try parseArguments(["changes"]))
XCTAssertThrowsError(try parseArguments(["changes", "--token", ""]))
}
func testUnknownCommandIsUsageError() {
XCTAssertThrowsError(try parseArguments([])) { error in
XCTAssertEqual((error as? HelperError)?.code, .usage)
XCTAssertEqual((error as? HelperError)?.exitCode, 64)
}
XCTAssertThrowsError(try parseArguments(["delete", "ID"]))
}
}

View File

@ -0,0 +1,92 @@
import XCTest
@testable import PhotosHelperCore
final class OutputTests: XCTestCase {
private func object(_ json: String) throws -> [String: Any] {
try XCTUnwrap(JSONSerialization.jsonObject(with: Data(json.utf8)) as? [String: Any])
}
func testItemEncodesEveryContractKeyWithExplicitNulls() throws {
let item = ClientMediaItem(
externalId: "ID/L0/001", kind: .photo, capturedAt: nil, durationSeconds: nil, width: 0,
height: nil, latitude: nil, longitude: nil, albumNames: [], isFavorite: false, fileName: nil)
let json = try object(encodeJSON(item))
XCTAssertEqual(
Set(json.keys),
[
"externalId", "kind", "capturedAt", "durationMs", "width", "height", "fps", "gpsLat",
"gpsLon", "deviceModel", "albumNames", "isFavorite", "fileName",
])
for key in ["capturedAt", "durationMs", "width", "height", "fps", "gpsLat", "gpsLon", "deviceModel", "fileName"] {
XCTAssertTrue(json[key] is NSNull, key)
}
}
func testItemNormalizesValues() throws {
let item = ClientMediaItem(
externalId: "V/L0/001", kind: .livePhoto, capturedAt: Date(timeIntervalSince1970: 1_704_164_645.5),
durationSeconds: 12.3456, width: 1920, height: 1080, latitude: 52.520_08, longitude: 13.404_95,
albumNames: ["Trip", "", String(repeating: "a", count: 300)] + Array(repeating: "x", count: 60),
isFavorite: true, fileName: "IMG_0001.HEIC")
let json = try object(encodeJSON(item))
XCTAssertEqual(json["kind"] as? String, "live_photo")
XCTAssertEqual(json["capturedAt"] as? String, "2024-01-02T03:04:05.500Z")
XCTAssertEqual(json["durationMs"] as? Int, 12_346)
XCTAssertEqual(json["gpsLat"] as? Double, 52.52)
XCTAssertEqual(json["gpsLon"] as? Double, 13.4)
let albums = try XCTUnwrap(json["albumNames"] as? [String])
XCTAssertEqual(albums.count, 50)
XCTAssertEqual(albums[1].count, 200)
}
func testInvalidCoordinatesAreDropped() {
XCTAssertNil(roundedCoordinate(latitude: 91, longitude: 0.5))
XCTAssertNil(roundedCoordinate(latitude: .nan, longitude: 1))
XCTAssertNil(roundedCoordinate(latitude: 0, longitude: 0))
XCTAssertNil(roundedCoordinate(latitude: 1, longitude: nil))
}
func testClipCountsUTF16Units() {
XCTAssertEqual(clip("abc", maxCharacters: 5), "abc")
// Each emoji is two UTF-16 units; a half character is never kept.
XCTAssertEqual(clip("😀😀😀", maxCharacters: 5), "😀😀")
}
func testErrorAndProgressLinesAreSingleLineJSON() throws {
let line = errorLine(HelperError(.notFound, "No \"asset\"\nhere."))
XCTAssertFalse(line.contains("\n"))
let error = try XCTUnwrap(try object(line)["error"] as? [String: String])
XCTAssertEqual(error["code"], "not_found")
XCTAssertEqual(error["message"], "No \"asset\"\nhere.")
for (input, expected) in [(0.42424, 0.424), (7, 1), (-1, 0), (.nan, 0)] as [(Double, Double)] {
let progress = progressLine(input)
XCTAssertFalse(progress.contains("\n"))
XCTAssertEqual(try object(progress)["progress"] as? Double ?? -1, expected, accuracy: 1e-9)
}
}
func testListAndChangesKeepNullKeys() throws {
let list = try object(encodeJSON(ListOutput(items: [], nextCursor: nil, changeToken: nil)))
XCTAssertTrue(list["nextCursor"] is NSNull)
XCTAssertTrue(list["changeToken"] is NSNull)
let changes = try object(encodeJSON(ChangesOutput.fullResync()))
XCTAssertTrue(changes["token"] is NSNull)
XCTAssertEqual(changes["fullResyncRequired"] as? Bool, true)
}
func testExitCodesAreDistinctPerFailureClass() {
let codes = HelperError.Code.allCases.map { HelperError($0, "").exitCode }
XCTAssertFalse(codes.contains(0))
XCTAssertEqual(HelperError(.notAuthorized, "").exitCode, 77)
}
func testSHA256StreamsFiles() throws {
let url = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
defer { try? FileManager.default.removeItem(at: url) }
try Data("abc".utf8).write(to: url)
let digest = try sha256Hex(of: url)
XCTAssertEqual(digest.hex, "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad")
XCTAssertEqual(digest.sizeBytes, 3)
}
}

View File

@ -0,0 +1,102 @@
import XCTest
@testable import PhotosHelperCore
final class PagingTests: XCTestCase {
func testCursorRoundTripsAndIsURLSafe() throws {
for key in [
SortKey(creationMillis: 1_700_000_000_123, localIdentifier: "9F2A/L0/001"),
SortKey(creationMillis: nil, localIdentifier: "undated+/=?"),
] {
let encoded = ListCursor(key: key).encoded()
XCTAssertFalse(encoded.contains { "+/=".contains($0) }, encoded)
XCTAssertEqual(try ListCursor.decode(encoded).key, key)
}
}
func testCursorRejectsGarbage() {
for value in ["", "!!!", base64URLEncode(Data("{\"v\":2,\"d\":1,\"id\":\"x\"}".utf8)),
base64URLEncode(Data("{\"v\":1,\"d\":1,\"id\":\"\"}".utf8)), base64URLEncode(Data("[]".utf8))]
{
XCTAssertThrowsError(try ListCursor.decode(value), value)
}
}
func testSortKeyPutsUndatedFirstAndBreaksTiesByIdentifier() {
let keys = [
SortKey(creationMillis: 2, localIdentifier: "a"),
SortKey(creationMillis: 1, localIdentifier: "b"),
SortKey(creationMillis: nil, localIdentifier: "z"),
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: nil, localIdentifier: "c"),
]
XCTAssertEqual(
keys.sorted().map { key -> String in
let date = key.creationMillis.map { millis in String(millis) } ?? "-"
return date + key.localIdentifier
},
["-c", "-z", "1a", "1b", "2a"])
}
/// Simulates PhotoKit: sorted by date, ties in arbitrary (here reversed) order.
private func photoKitOrder(_ keys: [SortKey]) -> [SortKey] {
keys.sorted { lhs, rhs in
if lhs.creationMillis == rhs.creationMillis { return lhs.localIdentifier > rhs.localIdentifier }
return lhs < rhs
}
}
private func listAll(_ source: [SortKey], limit: Int) -> [SortKey] {
var cursor: ListCursor?
var listed: [SortKey] = []
var pages = 0
while true {
let page = nextPage(count: source.count, key: { source[$0] }, after: cursor, limit: limit)
XCTAssertLessThanOrEqual(page.indices.count, limit)
listed += page.indices.map { source[$0] }
pages += 1
guard page.hasMore, let last = listed.last else { break }
// The cursor survives a round trip through its encoded form, like the app stores it.
cursor = try? ListCursor.decode(ListCursor(key: last).encoded())
if pages > source.count + 1 { XCTFail("paging did not terminate"); break }
}
return listed
}
func testPagesCoverEverythingOnceInStableOrder() {
var keys: [SortKey] = []
for index in 0..<57 {
// Many equal timestamps (bursts) and a few undated assets.
let millis: Int64? = index % 11 == 0 ? nil : Int64(index / 4)
keys.append(SortKey(creationMillis: millis, localIdentifier: String(format: "id-%03d", index)))
}
let source = photoKitOrder(keys)
for limit in [1, 3, 4, 10, 500] {
XCTAssertEqual(listAll(source, limit: limit), keys.sorted(), "limit \(limit)")
}
}
func testEmptyLibraryHasNoMorePages() {
let page = nextPage(count: 0, key: { _ in fatalError() }, after: nil, limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testCursorAfterLastItemReturnsNothing() {
let source = [SortKey(creationMillis: 1, localIdentifier: "a")]
let page = nextPage(
count: 1, key: { source[$0] }, after: ListCursor(key: source[0]), limit: 10)
XCTAssertEqual(page.indices, [])
XCTAssertFalse(page.hasMore)
}
func testDeletedCursorItemStillResumesAfterIt() {
let source = [
SortKey(creationMillis: 1, localIdentifier: "a"),
SortKey(creationMillis: 3, localIdentifier: "c"),
]
let gone = ListCursor(key: SortKey(creationMillis: 2, localIdentifier: "b"))
let page = nextPage(count: 2, key: { source[$0] }, after: gone, limit: 10)
XCTAssertEqual(page.indices, [1])
}
}

View File

@ -0,0 +1,83 @@
import XCTest
@testable import PhotosHelperCore
final class PathGuardTests: XCTestCase {
private var directory: URL!
override func setUpWithError() throws {
directory = FileManager.default.temporaryDirectory.appendingPathComponent(
"photos-helper-\(UUID().uuidString)", isDirectory: true)
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
}
override func tearDownWithError() throws {
try? FileManager.default.removeItem(at: directory)
}
private func resolved() -> URL { directory.resolvingSymlinksInPath().standardizedFileURL }
func testWritesDirectChildrenOfTheResolvedDirectory() throws {
let target = try resolveExportDestination(
directory: directory.path, fileName: "IMG_0001.MOV", temporarySuffix: "t")
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path)
XCTAssertEqual(target.finalURL.lastPathComponent, "IMG_0001.MOV")
XCTAssertEqual(target.temporaryURL.lastPathComponent, ".IMG_0001.MOV.t.partial")
}
func testRejectsRelativeAndTraversingDirectories() {
for path in ["relative/dir", "\(directory.path)/../x", "\(directory.path)/./", "/a/../b", ""] {
XCTAssertThrowsError(
try resolveExportDestination(directory: path, fileName: "a.jpg"), path
) { error in
XCTAssertEqual((error as? HelperError)?.code, .pathRejected)
}
}
}
func testRejectsMissingDirectoryAndFiles() throws {
XCTAssertThrowsError(
try resolveExportDestination(directory: directory.path + "/missing", fileName: "a.jpg"))
let file = directory.appendingPathComponent("file")
try Data().write(to: file)
XCTAssertThrowsError(try resolveExportDestination(directory: file.path, fileName: "a.jpg"))
}
func testHostileFileNamesStayInside() throws {
for name in ["../../etc/passwd", "/etc/passwd", "..", ".hidden", "a/b\\c:d", "\u{0}x", " "] {
let target = try resolveExportDestination(directory: directory.path, fileName: name)
XCTAssertEqual(target.finalURL.deletingLastPathComponent().path, resolved().path, name)
XCTAssertFalse(target.fileName.hasPrefix("."), name)
XCTAssertFalse(target.fileName.contains("/"), name)
}
XCTAssertEqual(sanitizeFileName("../../etc/passwd"), "_.._etc_passwd")
XCTAssertEqual(sanitizeFileName(""), "asset")
}
func testLongNamesKeepTheirExtension() {
let name = sanitizeFileName(String(repeating: "é", count: 300) + ".mov")
XCTAssertLessThanOrEqual(name.utf8.count, 200)
XCTAssertTrue(name.hasSuffix(".mov"))
}
func testNeverOverwritesAnExistingFileOrSymlink() throws {
try Data().write(to: directory.appendingPathComponent("taken.jpg"))
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "taken.jpg")) {
error in XCTAssertEqual((error as? HelperError)?.code, .destinationExists)
}
try FileManager.default.createSymbolicLink(
atPath: directory.appendingPathComponent("link.jpg").path, withDestinationPath: "/nonexistent")
XCTAssertThrowsError(try resolveExportDestination(directory: directory.path, fileName: "link.jpg"))
}
func testSymlinkedDirectoryResolvesToItsTarget() throws {
let link = directory.appendingPathComponent("link", isDirectory: true)
let real = directory.appendingPathComponent("real", isDirectory: true)
try FileManager.default.createDirectory(at: real, withIntermediateDirectories: false)
try FileManager.default.createSymbolicLink(at: link, withDestinationURL: real)
let target = try resolveExportDestination(directory: link.path, fileName: "a.jpg")
XCTAssertEqual(
target.finalURL.deletingLastPathComponent().path,
real.resolvingSymlinksInPath().standardizedFileURL.path)
}
}

View File

@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<!-- electron-builder's default hardened-runtime entitlements for Electron. -->
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<!-- Hardened runtime blocks Photos without this, for the app and the bundled photos-helper. -->
<key>com.apple.security.personal-information.photos-library</key>
<true/>
</dict>
</plist>

View File

@ -0,0 +1,161 @@
import { describe, expect, it, vi } from "vitest";
import type { MediaIpcEvent, MediaIpcOptions } from "./media-ipc.js";
import {
MEDIA_PHOTOS_CHANNELS,
outcomeForError,
parseSyncRequest,
registerMediaIpc,
} from "./media-ipc.js";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import type { PhotosSyncResult } from "./media-photos-sync.js";
import { PhotosSyncError } from "./media-photos-sync.js";
type Listener = (event: MediaIpcEvent, ...args: unknown[]) => unknown;
const result: PhotosSyncResult = {
indexed: 2,
created: 2,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: true,
};
function setup(overrides: Partial<MediaIpcOptions> = {}) {
const handlers = new Map<string, Listener>();
const helper = {
authorize: vi.fn(async () => "authorized" as const),
} as unknown as PhotosHelper;
const options: MediaIpcOptions = {
ipcMain: { handle: (channel, listener) => handlers.set(channel, listener) },
platform: "darwin",
helper,
isTrusted: () => true,
runSync: vi.fn(async () => result),
...overrides,
};
const registration = registerMediaIpc(options);
const sent: [string, unknown][] = [];
const event: MediaIpcEvent = {
sender: { send: (channel, payload) => sent.push([channel, payload]), isDestroyed: () => false },
};
const call = (channel: string, ...args: unknown[]) => handlers.get(channel)!(event, ...args);
return { handlers, helper, options, registration, call, sent };
}
describe("media IPC on platforms without Apple Photos", () => {
it.each(["linux", "win32"])(
"reports %s as unavailable and rejects sync and cancel",
async (platform) => {
const { call, options, handlers } = setup({ platform });
expect([...handlers.keys()].sort()).toEqual(
[
MEDIA_PHOTOS_CHANNELS.cancel,
MEDIA_PHOTOS_CHANNELS.status,
MEDIA_PHOTOS_CHANNELS.sync,
].sort(),
);
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: null })).toThrow(/macOS/);
expect(() => call(MEDIA_PHOTOS_CHANNELS.cancel)).toThrow(/macOS/);
expect(options.runSync).not.toHaveBeenCalled();
expect(options.helper!.authorize).not.toHaveBeenCalled();
},
);
it("treats a macOS build without the helper as unavailable", async () => {
const { call } = setup({ helper: null });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow(/macOS/);
});
it("treats a helper that cannot start as unavailable", async () => {
const { call, helper } = setup();
vi.mocked(helper.authorize).mockRejectedValueOnce(new PhotosHelperError("unavailable", "x"));
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({ available: false });
});
});
describe("media IPC on macOS", () => {
it("reads status without prompting", async () => {
const { call, helper } = setup();
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toEqual({
available: true,
authorization: "authorized",
syncing: false,
});
expect(helper.authorize).toHaveBeenCalledWith(false);
});
it("runs one sync at a time, forwards progress and resolves with the outcome", async () => {
let finish: (value: PhotosSyncResult) => void = () => {};
const runSync = vi.fn(
({ onProgress }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((resolve) => {
onProgress({ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 });
finish = resolve;
}),
);
const { call, sent } = setup({ runSync });
const first = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
const second = call(MEDIA_PHOTOS_CHANNELS.sync, { spaceId: "space_1" });
expect(second).toBe(first);
expect(runSync).toHaveBeenCalledTimes(1);
expect(runSync.mock.calls[0]?.[0].spaceId).toBe("space_1");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: true });
expect(sent).toEqual([
[
MEDIA_PHOTOS_CHANNELS.progress,
{ phase: "indexing", indexed: 1, uploadsDone: 0, uploadsTotal: 0 },
],
]);
finish(result);
await expect(first).resolves.toEqual({ ok: true, result });
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).resolves.toMatchObject({ syncing: false });
});
it("cancels the running sync", async () => {
const runSync = vi.fn(
({ signal }: Parameters<MediaIpcOptions["runSync"]>[0]) =>
new Promise<PhotosSyncResult>((_resolve, reject) => {
signal.addEventListener("abort", () =>
reject(new PhotosSyncError("cancelled", "Sync cancelled.")),
);
}),
);
const { call } = setup({ runSync });
const running = call(MEDIA_PHOTOS_CHANNELS.sync, {});
await call(MEDIA_PHOTOS_CHANNELS.cancel);
await expect(running).resolves.toEqual({
ok: false,
code: "cancelled",
message: "Sync cancelled.",
});
});
it("rejects untrusted senders and malformed requests", async () => {
const { call } = setup({ isTrusted: () => false });
expect(() => call(MEDIA_PHOTOS_CHANNELS.sync, {})).toThrow("Not allowed.");
await expect(call(MEDIA_PHOTOS_CHANNELS.status)).rejects.toThrow("Not allowed.");
expect(() => parseSyncRequest(null)).toThrow();
expect(() => parseSyncRequest({ spaceId: "../x" })).toThrow();
expect(() => parseSyncRequest({ spaceId: 4 })).toThrow();
expect(parseSyncRequest({})).toEqual({ spaceId: null });
});
it("never forwards unexpected error text to the renderer", () => {
expect(outcomeForError(new Error("/Users/example/secret path"))).toEqual({
ok: false,
code: "failed",
message: "Apple Photos sync failed.",
});
expect(
outcomeForError(
new PhotosHelperError("not_authorized", "Photos access has not been granted."),
),
).toMatchObject({
code: "not_authorized",
});
});
});

View File

@ -0,0 +1,135 @@
import type { PhotosAuthorization, PhotosHelper } from "./media-photos-helper.js";
import { PhotosHelperError } from "./media-photos-helper.js";
import type { PhotosSyncProgress, PhotosSyncResult } from "./media-photos-sync.js";
import { PhotosSyncError } from "./media-photos-sync.js";
/*
* IPC for the Apple Photos source. The renderer only expresses intent (sync, cancel)
* and renders state; the main process owns the helper, the session and the upload.
*/
export const MEDIA_PHOTOS_CHANNELS = {
status: "desktop.media.photos.status",
sync: "desktop.media.photos.sync",
cancel: "desktop.media.photos.cancel",
progress: "desktop.media.photos.progress",
} as const;
export type PhotosSourceStatus =
| { available: false }
| { available: true; authorization: PhotosAuthorization; syncing: boolean };
export type PhotosSyncOutcome =
| { ok: true; result: PhotosSyncResult }
| { ok: false; code: string; message: string };
/** Renderer-facing bridge, exposed as `rakazoDesktop.media.photos`. */
export interface RakazoDesktopMediaPhotos {
status: () => Promise<PhotosSourceStatus>;
/** Asks for Photos access on first use, then indexes and uploads what the server needs. */
sync: (request: { spaceId: string | null }) => Promise<PhotosSyncOutcome>;
cancel: () => Promise<void>;
onProgress: (listener: (progress: PhotosSyncProgress) => void) => () => void;
}
export interface MediaIpcEvent {
sender: { send(channel: string, payload: unknown): void; isDestroyed(): boolean };
}
export interface MediaIpcMain {
handle(channel: string, listener: (event: MediaIpcEvent, ...args: unknown[]) => unknown): void;
}
export interface MediaIpcOptions {
ipcMain: MediaIpcMain;
platform: string;
/** Null when the helper binary is missing (unbuilt dev checkout). */
helper: PhotosHelper | null;
/** Same gate as the other app-window handlers: main window, main frame. */
isTrusted: (event: MediaIpcEvent) => boolean;
runSync: (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}) => Promise<PhotosSyncResult>;
}
const UNAVAILABLE = "Apple Photos is only available in the macOS app.";
const SPACE_ID = /^[A-Za-z0-9_-]{1,200}$/;
export function parseSyncRequest(value: unknown): { spaceId: string | null } {
if (typeof value !== "object" || value === null) throw new Error("Invalid sync request.");
const spaceId = (value as { spaceId?: unknown }).spaceId ?? null;
if (spaceId !== null && (typeof spaceId !== "string" || !SPACE_ID.test(spaceId))) {
throw new Error("Invalid sync request.");
}
return { spaceId };
}
/** Registers the handlers; `cancel()` stops a running sync, e.g. before quit. */
export function registerMediaIpc(options: MediaIpcOptions): { cancel: () => void } {
const available = options.platform === "darwin" && options.helper !== null;
let running: { controller: AbortController; done: Promise<PhotosSyncOutcome> } | null = null;
const requireTrusted = (event: MediaIpcEvent) => {
if (!options.isTrusted(event)) throw new Error("Not allowed.");
};
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.status, async (event) => {
requireTrusted(event);
if (!available) return { available: false } satisfies PhotosSourceStatus;
try {
const authorization = await options.helper!.authorize(false);
return { available: true, authorization, syncing: running !== null };
} catch (error) {
if (error instanceof PhotosHelperError && error.code === "unavailable") {
return { available: false } satisfies PhotosSourceStatus;
}
throw error;
}
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.sync, (event, request: unknown) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
const { spaceId } = parseSyncRequest(request);
// A second click joins the running sync instead of starting a parallel one.
if (running) return running.done;
const controller = new AbortController();
const done = options
.runSync({
spaceId,
signal: controller.signal,
onProgress: (progress) => {
if (!event.sender.isDestroyed()) {
event.sender.send(MEDIA_PHOTOS_CHANNELS.progress, progress);
}
},
})
.then(
(result): PhotosSyncOutcome => ({ ok: true, result }),
(error: unknown): PhotosSyncOutcome => outcomeForError(error),
)
.finally(() => {
if (running?.controller === controller) running = null;
});
running = { controller, done };
return done;
});
options.ipcMain.handle(MEDIA_PHOTOS_CHANNELS.cancel, (event) => {
requireTrusted(event);
if (!available) throw new Error(UNAVAILABLE);
running?.controller.abort();
});
return { cancel: () => running?.controller.abort() };
}
/** Only our own fixed messages reach the renderer; anything else is generic. */
export function outcomeForError(error: unknown): PhotosSyncOutcome {
if (error instanceof PhotosSyncError || error instanceof PhotosHelperError) {
return { ok: false, code: error.code, message: error.message };
}
return { ok: false, code: "failed", message: "Apple Photos sync failed." };
}

View File

@ -0,0 +1,437 @@
import { EventEmitter } from "node:events";
import { mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { PassThrough } from "node:stream";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { HelperProcess, SpawnHelper } from "./media-photos-helper.js";
import {
changesArgs,
createPhotosHelper,
exportArgs,
helperEnv,
listArgs,
PhotosHelperError,
parseAssetOutput,
parseAuthorizeOutput,
parseChangesOutput,
parseExportOutput,
parseHelperError,
parseListOutput,
parseProgressLine,
photosHelperCandidates,
resolvePhotosHelperBinary,
runPhotosHelper,
} from "./media-photos-helper.js";
const item = (overrides: Record<string, unknown> = {}) => ({
externalId: "9F2A61B0-0000-4000-8000-000000000001/L0/001",
kind: "video",
capturedAt: "2024-01-02T03:04:05.500Z",
durationMs: 12_346,
width: 1920,
height: 1080,
fps: null,
gpsLat: 52.52,
gpsLon: 13.4,
deviceModel: null,
albumNames: ["Trip"],
isFavorite: true,
fileName: "IMG_0001.MOV",
...overrides,
});
const listJson = (items: unknown[], nextCursor: unknown = null, changeToken: unknown = "tok") =>
JSON.stringify({ items, nextCursor, changeToken });
function expectCode(run: () => unknown, code: string) {
try {
run();
} catch (error) {
expect(error).toBeInstanceOf(PhotosHelperError);
expect((error as PhotosHelperError).code).toBe(code);
return;
}
throw new Error("expected a PhotosHelperError");
}
describe("photos helper output parsing", () => {
it("accepts a list page that matches the shared item contract", () => {
const page = parseListOutput(
listJson([item(), item({ kind: "photo", durationMs: null })], "c1"),
2,
);
expect(page.nextCursor).toBe("c1");
expect(page.changeToken).toBe("tok");
expect(page.items).toHaveLength(2);
expect(page.items[0]).toEqual(item());
});
it("accepts the last page of an empty library", () => {
expect(parseListOutput(listJson([], null, null), 500)).toEqual({
items: [],
nextCursor: null,
changeToken: null,
});
});
it.each([
["not JSON", "items: []"],
["an array document", "[]"],
["truncated JSON", '{"items":[{"externalId":'],
["missing items", JSON.stringify({ nextCursor: null, changeToken: null })],
["an item missing a key", listJson([{ ...item(), width: undefined }])],
["an unknown kind", listJson([item({ kind: "audio" })])],
["out-of-range GPS", listJson([item({ gpsLat: 91 })])],
["a non-UTC-parsable date", listJson([item({ capturedAt: "yesterday" })])],
["a negative duration", listJson([item({ durationMs: -1 })])],
["an empty external id", listJson([item({ externalId: "" })])],
["an empty cursor", listJson([item()], "")],
["a numeric cursor", listJson([item()], 3)],
["too many items", listJson([item(), item(), item()])],
])("rejects %s", (_name, text) => {
expectCode(() => parseListOutput(text, 2), "invalid_output");
});
it("parses authorization states and rejects unknown ones", () => {
expect(parseAuthorizeOutput('{"status":"limited"}')).toBe("limited");
expect(parseAuthorizeOutput('{"status":"notDetermined"}')).toBe("notDetermined");
expectCode(() => parseAuthorizeOutput('{"status":"maybe"}'), "invalid_output");
expectCode(() => parseAuthorizeOutput(""), "invalid_output");
});
it("parses asset details", () => {
const detail = parseAssetOutput(
JSON.stringify({
item: item(),
modifiedAt: "2024-02-01T00:00:00.000Z",
hidden: false,
resources: [{ type: "video", fileName: "IMG_0001.MOV", uti: "com.apple.quicktime-movie" }],
}),
);
expect(detail.resources[0]?.type).toBe("video");
expectCode(
() => parseAssetOutput(JSON.stringify({ item: item(), modifiedAt: null, resources: [] })),
"invalid_output",
);
});
it("parses changes and requires a token unless a full relisting is needed", () => {
expect(
parseChangesOutput(
JSON.stringify({
token: "t2",
changed: [item()],
deleted: ["x/L0/001"],
fullResyncRequired: false,
}),
),
).toMatchObject({ token: "t2", deleted: ["x/L0/001"], fullResyncRequired: false });
expect(
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: true }),
).fullResyncRequired,
).toBe(true);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: null, changed: [], deleted: [], fullResyncRequired: false }),
),
"invalid_output",
);
expectCode(
() =>
parseChangesOutput(
JSON.stringify({ token: "t", changed: [], deleted: [7], fullResyncRequired: false }),
),
"invalid_output",
);
});
it("maps helper error lines to codes", () => {
const stderr =
'{"progress":0.5}\n{"error":{"code":"not_authorized","message":"Photos access has not been granted."}}\n';
const error = parseHelperError(stderr, 77);
expect(error.code).toBe("not_authorized");
expect(error.message).toBe("Photos access has not been granted.");
expect(parseHelperError('{"error":{"code":"internal","message":"x"}}', 70).code).toBe("failed");
expect(parseHelperError("Segmentation fault", 139).message).toContain("139");
expect(parseHelperError('{"error":"oops"}', 1).code).toBe("failed");
});
it("reads progress lines only", () => {
expect(parseProgressLine('{"progress":0.42}')).toBe(0.42);
expect(parseProgressLine('{"progress":2}')).toBeNull();
expect(parseProgressLine('{"error":{"code":"usage"}}')).toBeNull();
expect(parseProgressLine('{"progress":')).toBeNull();
});
});
describe("photos helper export result", () => {
let dir: string;
let other: string;
beforeEach(async () => {
dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-export-"));
other = await mkdtemp(path.join(tmpdir(), "rakazo-photos-other-"));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
await rm(other, { recursive: true, force: true });
});
const result = (file: string, fileName = path.basename(file)) =>
JSON.stringify({ path: file, fileName, sizeBytes: 3, sha256: "a".repeat(64) });
it("accepts a file directly inside the export folder", async () => {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
await expect(parseExportOutput(result(file), dir)).resolves.toMatchObject({
fileName: "IMG_0001.MOV",
sizeBytes: 3,
});
});
it("rejects files outside the export folder, including through symlinks", async () => {
const outside = path.join(other, "secret.txt");
await writeFile(outside, "abc");
await expect(parseExportOutput(result(outside), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const link = path.join(dir, "link.MOV");
await symlink(outside, link);
await expect(parseExportOutput(result(link), dir)).rejects.toMatchObject({
code: "path_rejected",
});
const nested = path.join(dir, "..", path.basename(other), "secret.txt");
await expect(parseExportOutput(result(nested), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
it("rejects malformed results", async () => {
const file = path.join(dir, "a.jpg");
await writeFile(file, "abc");
for (const text of [
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: 3, sha256: "XYZ" }),
JSON.stringify({ path: "a.jpg", fileName: "a.jpg", sizeBytes: 3, sha256: "a".repeat(64) }),
JSON.stringify({ path: file, fileName: "a.jpg", sizeBytes: -1, sha256: "a".repeat(64) }),
JSON.stringify({
path: path.join(dir, "missing.jpg"),
fileName: "missing.jpg",
sizeBytes: 3,
sha256: "a".repeat(64),
}),
]) {
await expect(parseExportOutput(text, dir)).rejects.toMatchObject({ code: "invalid_output" });
}
await expect(parseExportOutput(result(file, "b.jpg"), dir)).rejects.toMatchObject({
code: "path_rejected",
});
});
});
describe("photos helper argv", () => {
it("builds fixed argv arrays", () => {
expect(listArgs({ cursor: null, limit: 500 })).toEqual(["list", "--limit", "500"]);
expect(listArgs({ cursor: "abc_-", limit: 10, since: "2024-01-01T00:00:00Z" })).toEqual([
"list",
"--limit",
"10",
"--cursor",
"abc_-",
"--since",
"2024-01-01T00:00:00Z",
]);
expect(exportArgs("ID/L0/001", "/tmp/rakazo-photos-1")).toEqual([
"export",
"ID/L0/001",
"--dest",
"/tmp/rakazo-photos-1",
"--original",
]);
expect(changesArgs("tok")).toEqual(["changes", "--token", "tok"]);
});
it("keeps shell syntax as inert data and rejects option-like values", () => {
// No shell runs, so metacharacters stay one literal argument.
expect(exportArgs("$(rm -rf ~);`x`", "/tmp/a b")[1]).toBe("$(rm -rf ~);`x`");
expectCode(() => exportArgs("--dest", "/tmp"), "usage");
expectCode(() => listArgs({ cursor: "--limit", limit: 1 }), "usage");
expectCode(() => changesArgs("a\nb"), "usage");
expectCode(() => listArgs({ cursor: null, limit: 501 }), "usage");
expectCode(() => exportArgs("ID", "relative/dir"), "path_rejected");
expectCode(() => exportArgs("ID", "/tmp/../etc"), "path_rejected");
});
it("passes only the environment PhotoKit needs", () => {
expect(
helperEnv({ HOME: "/Users/example", TMPDIR: "/tmp", OPENAI_API_KEY: "secret", PATH: "/bin" }),
).toEqual({ HOME: "/Users/example", TMPDIR: "/tmp" });
});
});
describe("photos helper binary", () => {
it("is only available on macOS", () => {
const options = {
packaged: true,
resourcesPath: "/App/Resources",
appPath: "/app",
exists: () => true,
};
expect(resolvePhotosHelperBinary({ ...options, platform: "linux" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "win32" })).toBeNull();
expect(resolvePhotosHelperBinary({ ...options, platform: "darwin" })).toBe(
"/App/Resources/bin/photos-helper",
);
});
it("falls back to the development build and reports a missing build", () => {
const options = {
platform: "darwin",
packaged: false,
resourcesPath: "/r",
appPath: "/repo/apps/desktop",
};
expect(photosHelperCandidates(options)).toEqual([
"/repo/apps/desktop/native/photos-helper/.build/apple/Products/Release/photos-helper",
"/repo/apps/desktop/native/photos-helper/.build/release/photos-helper",
]);
expect(
resolvePhotosHelperBinary({
...options,
exists: (file) => file.endsWith(".build/release/photos-helper"),
}),
).toBe("/repo/apps/desktop/native/photos-helper/.build/release/photos-helper");
expect(resolvePhotosHelperBinary({ ...options, exists: () => false })).toBeNull();
});
});
class FakeProcess extends EventEmitter implements HelperProcess {
stdout = new PassThrough();
stderr = new PassThrough();
kill = vi.fn(() => {
queueMicrotask(() => this.emit("close", null));
return true;
});
exit(code: number, stdout = "", stderr = "") {
if (stdout) this.stdout.write(stdout);
if (stderr) this.stderr.write(stderr);
setImmediate(() => this.emit("close", code));
}
}
function fakeSpawn() {
const processes: FakeProcess[] = [];
const spawn = vi.fn<SpawnHelper>(() => {
const child = new FakeProcess();
processes.push(child);
return child;
});
return { spawn, processes };
}
describe("photos helper process", () => {
it("spawns the binary with argv and no shell", async () => {
const { spawn, processes } = fakeSpawn();
const helper = createPhotosHelper("/App/Resources/bin/photos-helper", spawn);
const pending = helper.list({ cursor: null, limit: 2 });
processes[0]!.exit(0, listJson([item()]));
await expect(pending).resolves.toMatchObject({ items: [item()] });
expect(spawn).toHaveBeenCalledWith(
"/App/Resources/bin/photos-helper",
["list", "--limit", "2"],
expect.objectContaining({ shell: false, stdio: ["ignore", "pipe", "pipe"] }),
);
});
it("maps a failing exit to the helper's error code", async () => {
const { spawn, processes } = fakeSpawn();
const pending = createPhotosHelper("/bin/photos-helper", spawn).asset("ID/L0/001");
processes[0]!.exit(
66,
"",
'{"error":{"code":"not_found","message":"No photo or video with that identifier."}}\n',
);
await expect(pending).rejects.toMatchObject({ code: "not_found" });
});
it("reports export progress and validates the result", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-run-"));
try {
const file = path.join(dir, "IMG_0001.MOV");
await writeFile(file, "abc");
const { spawn, processes } = fakeSpawn();
const progress: number[] = [];
const pending = createPhotosHelper("/bin/photos-helper", spawn).exportOriginal("ID", dir, {
onProgress: (value) => progress.push(value),
});
const child = processes[0]!;
child.stderr.write('{"progress":0}\n{"progress":0.5');
child.stderr.write('}\nnoise\n{"progress":1}\n');
child.exit(
0,
JSON.stringify({
path: file,
fileName: "IMG_0001.MOV",
sizeBytes: 3,
sha256: "b".repeat(64),
}),
);
await expect(pending).resolves.toMatchObject({ sizeBytes: 3 });
expect(progress).toEqual([0, 0.5, 1]);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("kills the helper on abort and on timeout", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
const aborted = runPhotosHelper("/bin/h", ["list"], {
spawn,
timeoutMs: 60_000,
signal: controller.signal,
});
controller.abort();
await expect(aborted).rejects.toMatchObject({ code: "cancelled" });
expect(processes[0]!.kill).toHaveBeenCalledWith("SIGTERM");
const timedOut = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 5 });
await expect(timedOut).rejects.toMatchObject({ code: "timeout" });
expect(processes[1]!.kill).toHaveBeenCalled();
});
it("fails a silent export through the idle limit", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["export"], {
spawn,
timeoutMs: 60_000,
idleTimeoutMs: 5,
});
await expect(pending).rejects.toMatchObject({ code: "timeout" });
expect(processes[0]!.kill).toHaveBeenCalled();
});
it("does not start when already aborted and reports spawn failures", async () => {
const { spawn, processes } = fakeSpawn();
const controller = new AbortController();
controller.abort();
await expect(
runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000, signal: controller.signal }),
).rejects.toMatchObject({ code: "cancelled" });
expect(spawn).not.toHaveBeenCalled();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.emit("error", new Error("ENOENT"));
await expect(pending).rejects.toMatchObject({ code: "unavailable" });
});
it("rejects oversized output", async () => {
const { spawn, processes } = fakeSpawn();
const pending = runPhotosHelper("/bin/h", ["list"], { spawn, timeoutMs: 1000 });
processes[0]!.stdout.write(Buffer.alloc(33 * 1024 * 1024));
await expect(pending).rejects.toMatchObject({ code: "invalid_output" });
});
});

View File

@ -0,0 +1,517 @@
import { spawn as nodeSpawn } from "node:child_process";
import { realpath } from "node:fs/promises";
import path from "node:path";
import type { Readable } from "node:stream";
import type { ClientMediaItem } from "@rakazo/contracts";
import { ClientMediaItemSchema } from "@rakazo/contracts";
/*
* Client for the macOS `photos-helper` CLI (apps/desktop/native/photos-helper). The
* helper runs with a fixed argv and no shell; every byte it prints is validated before
* the app uses it, so a malformed or unexpected answer fails instead of being guessed at.
*/
export type PhotosAuthorization =
| "authorized"
| "limited"
| "denied"
| "restricted"
| "notDetermined";
export interface PhotosListPage {
items: ClientMediaItem[];
nextCursor: string | null;
/** Library change token from before this page was read. */
changeToken: string | null;
}
export interface PhotosAssetDetail {
item: ClientMediaItem;
modifiedAt: string | null;
hidden: boolean;
resources: { type: string; fileName: string; uti: string }[];
}
export interface PhotosChanges {
token: string | null;
changed: ClientMediaItem[];
deleted: string[];
fullResyncRequired: boolean;
}
export interface PhotosExport {
path: string;
fileName: string;
sizeBytes: number;
sha256: string;
}
export type PhotosHelperErrorCode =
| "unavailable"
| "usage"
| "not_authorized"
| "not_found"
| "path_rejected"
| "destination_exists"
| "network"
| "export_failed"
| "invalid_output"
| "timeout"
| "cancelled"
| "failed";
const HELPER_CODES = new Set<PhotosHelperErrorCode>([
"usage",
"not_authorized",
"not_found",
"path_rejected",
"destination_exists",
"network",
"export_failed",
]);
export class PhotosHelperError extends Error {
constructor(
readonly code: PhotosHelperErrorCode,
message: string,
) {
super(message);
this.name = "PhotosHelperError";
}
}
export const PHOTOS_HELPER_NAME = "photos-helper";
/** Matches the contract's batch limit for `media.sources.pushItems`. */
export const PHOTOS_MAX_PAGE = 500;
/**
* Packaged apps ship the universal helper in Resources/bin; development uses the
* SwiftPM output (universal `--arch` build first, then the host-arch build).
*/
export function photosHelperCandidates(options: {
packaged: boolean;
resourcesPath: string;
appPath: string;
}): string[] {
if (options.packaged) return [path.join(options.resourcesPath, "bin", PHOTOS_HELPER_NAME)];
const build = path.join(options.appPath, "native", "photos-helper", ".build");
return [
path.join(build, "apple", "Products", "Release", PHOTOS_HELPER_NAME),
path.join(build, "release", PHOTOS_HELPER_NAME),
];
}
/** Null off macOS or when the helper was not built; the Apple Photos source is then hidden. */
export function resolvePhotosHelperBinary(options: {
platform: string;
packaged: boolean;
resourcesPath: string;
appPath: string;
exists: (file: string) => boolean;
}): string | null {
if (options.platform !== "darwin") return null;
return photosHelperCandidates(options).find((candidate) => options.exists(candidate)) ?? null;
}
// ---------------------------------------------------------------------------
// argv
// ---------------------------------------------------------------------------
export function authorizeArgs(prompt: boolean): string[] {
return prompt ? ["authorize"] : ["authorize", "--no-prompt"];
}
export function listArgs(request: {
cursor: string | null;
limit: number;
since?: string | null;
}): string[] {
if (!Number.isInteger(request.limit) || request.limit < 1 || request.limit > PHOTOS_MAX_PAGE) {
throw new PhotosHelperError("usage", "List limit is out of range.");
}
const args = ["list", "--limit", String(request.limit)];
if (request.cursor !== null) args.push("--cursor", opaqueArgument(request.cursor));
if (request.since) args.push("--since", opaqueArgument(request.since));
return args;
}
export function assetArgs(localIdentifier: string): string[] {
return ["asset", identifierArgument(localIdentifier)];
}
export function exportArgs(localIdentifier: string, destination: string): string[] {
if (!path.isAbsolute(destination) || destination.split(/[\\/]/).includes("..")) {
throw new PhotosHelperError("path_rejected", "Export directory must be absolute.");
}
return ["export", identifierArgument(localIdentifier), "--dest", destination, "--original"];
}
export function changesArgs(token: string): string[] {
return ["changes", "--token", opaqueArgument(token)];
}
/** Identifiers come from the helper itself; a leading dash would read as an option. */
function identifierArgument(value: string): string {
if (value === "" || value.length > 200 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid Photos identifier.");
}
return value;
}
function opaqueArgument(value: string): string {
if (value === "" || value.length > 4096 || value.startsWith("-") || hasControl(value)) {
throw new PhotosHelperError("usage", "Invalid helper argument.");
}
return value;
}
function hasControl(value: string): boolean {
for (const char of value) {
const code = char.codePointAt(0) ?? 0;
if (code < 0x20 || code === 0x7f) return true;
}
return false;
}
// ---------------------------------------------------------------------------
// Output parsing
// ---------------------------------------------------------------------------
function invalid(what: string): never {
throw new PhotosHelperError("invalid_output", `Photos helper returned invalid ${what}.`);
}
function parseDocument(text: string): Record<string, unknown> {
let value: unknown;
try {
value = JSON.parse(text);
} catch {
invalid("JSON");
}
if (typeof value !== "object" || value === null || Array.isArray(value)) invalid("JSON");
return value as Record<string, unknown>;
}
function nullableString(value: unknown, what: string, max = 4096): string | null {
if (value === null) return null;
if (typeof value !== "string" || value === "" || value.length > max) invalid(what);
return value;
}
/** Each item must satisfy the shared contract schema exactly as the server will parse it. */
function parseItems(value: unknown, what: string): ClientMediaItem[] {
if (!Array.isArray(value) || value.length > 10_000) invalid(what);
return value.map((entry) => {
if (typeof entry !== "object" || entry === null) invalid(what);
const parsed = ClientMediaItemSchema.safeParse(entry);
if (!parsed.success) invalid(what);
return parsed.data;
});
}
const AUTHORIZATION = new Set<PhotosAuthorization>([
"authorized",
"limited",
"denied",
"restricted",
"notDetermined",
]);
export function parseAuthorizeOutput(text: string): PhotosAuthorization {
const status = parseDocument(text).status;
if (typeof status !== "string" || !AUTHORIZATION.has(status as PhotosAuthorization)) {
invalid("authorization status");
}
return status as PhotosAuthorization;
}
export function parseListOutput(text: string, limit: number): PhotosListPage {
const document = parseDocument(text);
const items = parseItems(document.items, "list items");
if (items.length > limit) invalid("list page size");
const nextCursor = nullableString(document.nextCursor, "list cursor");
const changeToken = nullableString(document.changeToken, "change token");
return { items, nextCursor, changeToken };
}
export function parseAssetOutput(text: string): PhotosAssetDetail {
const document = parseDocument(text);
const [item] = parseItems([document.item], "asset");
const modifiedAt = nullableString(document.modifiedAt, "asset date", 64);
if (modifiedAt !== null && Number.isNaN(Date.parse(modifiedAt))) invalid("asset date");
if (typeof document.hidden !== "boolean") invalid("asset");
if (!Array.isArray(document.resources) || document.resources.length > 50) invalid("resources");
const resources = document.resources.map((resource) => {
if (typeof resource !== "object" || resource === null) invalid("resources");
const { type, fileName, uti } = resource as Record<string, unknown>;
if (typeof type !== "string" || typeof fileName !== "string" || typeof uti !== "string") {
invalid("resources");
}
return { type, fileName, uti };
});
return { item: item!, modifiedAt, hidden: document.hidden, resources };
}
export function parseChangesOutput(text: string): PhotosChanges {
const document = parseDocument(text);
const token = nullableString(document.token, "change token");
if (typeof document.fullResyncRequired !== "boolean") invalid("changes");
const changed = parseItems(document.changed, "changed items");
if (!Array.isArray(document.deleted) || document.deleted.length > 10_000) invalid("changes");
const deleted = document.deleted.map((id) => {
if (typeof id !== "string" || id === "" || id.length > 200) invalid("deleted ids");
return id;
});
if (!document.fullResyncRequired && token === null) invalid("change token");
return { token, changed, deleted, fullResyncRequired: document.fullResyncRequired };
}
/**
* Besides the shape, the file must be a direct child of the directory the app chose,
* after resolving symlinks, so a confused helper cannot point the upload elsewhere.
*/
export async function parseExportOutput(text: string, destination: string): Promise<PhotosExport> {
const document = parseDocument(text);
const { path: file, fileName, sizeBytes, sha256 } = document;
if (
typeof file !== "string" ||
!path.isAbsolute(file) ||
typeof fileName !== "string" ||
fileName === "" ||
typeof sizeBytes !== "number" ||
!Number.isSafeInteger(sizeBytes) ||
sizeBytes < 0 ||
typeof sha256 !== "string" ||
!/^[0-9a-f]{64}$/.test(sha256)
) {
invalid("export result");
}
let real: string;
let realDestination: string;
try {
[real, realDestination] = await Promise.all([realpath(file), realpath(destination)]);
} catch {
invalid("export path");
}
if (path.dirname(real) !== realDestination || path.basename(real) !== fileName) {
throw new PhotosHelperError("path_rejected", "Photos helper wrote outside the export folder.");
}
return { path: real, fileName, sizeBytes, sha256 };
}
/** The last `{"error":{code,message}}` line on stderr; unknown codes become `failed`. */
export function parseHelperError(stderr: string, exitCode: number | null): PhotosHelperError {
const lines = stderr.split("\n").reverse();
for (const line of lines) {
if (!line.startsWith("{")) continue;
try {
const error = (JSON.parse(line) as { error?: unknown }).error as
| { code?: unknown; message?: unknown }
| undefined;
if (typeof error?.code !== "string") continue;
const code = HELPER_CODES.has(error.code as PhotosHelperErrorCode)
? (error.code as PhotosHelperErrorCode)
: "failed";
const message =
typeof error.message === "string" && error.message.length <= 500
? error.message
: "Photos helper failed.";
return new PhotosHelperError(code, message);
} catch {}
}
return new PhotosHelperError("failed", `Photos helper exited with status ${exitCode ?? "?"}.`);
}
/** `{"progress":0.42}` lines on stderr; anything else is ignored. */
export function parseProgressLine(line: string): number | null {
if (!line.startsWith('{"progress"')) return null;
try {
const value = (JSON.parse(line) as { progress?: unknown }).progress;
return typeof value === "number" && value >= 0 && value <= 1 ? value : null;
} catch {
return null;
}
}
// ---------------------------------------------------------------------------
// Process
// ---------------------------------------------------------------------------
/** The subset of ChildProcess the runner needs, so tests can supply a fake. */
export interface HelperProcess {
stdout: Readable | null;
stderr: Readable | null;
kill(signal?: NodeJS.Signals): boolean;
on(event: "error", listener: (error: Error) => void): unknown;
on(event: "close", listener: (code: number | null) => void): unknown;
}
export type SpawnHelper = (
binary: string,
args: string[],
options: { shell: false; env: Record<string, string>; stdio: ["ignore", "pipe", "pipe"] },
) => HelperProcess;
export interface RunOptions {
signal?: AbortSignal;
/** Hard limit for the whole run. */
timeoutMs: number;
/** Fails when stderr progress stops for this long (exports only). */
idleTimeoutMs?: number;
onProgress?: (fraction: number) => void;
}
const MAX_STDOUT_BYTES = 32 * 1024 * 1024;
const MAX_STDERR_BYTES = 1024 * 1024;
const ENV_ALLOWLIST = ["HOME", "USER", "LOGNAME", "TMPDIR", "LANG", "LC_ALL"];
/** PhotoKit needs the user's home; nothing else from the app environment is passed on. */
export function helperEnv(env: NodeJS.ProcessEnv): Record<string, string> {
const result: Record<string, string> = {};
for (const name of ENV_ALLOWLIST) {
const value = env[name];
if (value) result[name] = value;
}
return result;
}
export const defaultSpawnHelper: SpawnHelper = (binary, args, options) =>
nodeSpawn(binary, args, options);
export function runPhotosHelper(
binary: string,
args: string[],
options: RunOptions & { spawn?: SpawnHelper; env?: NodeJS.ProcessEnv },
): Promise<string> {
return new Promise((resolve, reject) => {
if (options.signal?.aborted) {
reject(new PhotosHelperError("cancelled", "Cancelled."));
return;
}
const child = (options.spawn ?? defaultSpawnHelper)(binary, args, {
shell: false,
env: helperEnv(options.env ?? process.env),
stdio: ["ignore", "pipe", "pipe"],
});
const stdout: Buffer[] = [];
let stdoutBytes = 0;
let stderr = "";
let pendingLine = "";
let settled = false;
let idleTimer: NodeJS.Timeout | undefined;
const finish = (error: PhotosHelperError | null, output = "") => {
if (settled) return;
settled = true;
clearTimeout(timer);
clearTimeout(idleTimer);
options.signal?.removeEventListener("abort", abort);
if (error) reject(error);
else resolve(output);
};
const stop = (error: PhotosHelperError) => {
child.kill("SIGTERM");
finish(error);
};
const abort = () => stop(new PhotosHelperError("cancelled", "Cancelled."));
const timer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos helper timed out.")),
options.timeoutMs,
);
timer.unref?.();
const armIdle = () => {
if (options.idleTimeoutMs === undefined) return;
clearTimeout(idleTimer);
idleTimer = setTimeout(
() => stop(new PhotosHelperError("timeout", "Photos export stalled.")),
options.idleTimeoutMs,
);
idleTimer.unref?.();
};
armIdle();
options.signal?.addEventListener("abort", abort, { once: true });
child.stdout?.on("data", (chunk: Buffer) => {
stdoutBytes += chunk.byteLength;
if (stdoutBytes > MAX_STDOUT_BYTES) {
stop(new PhotosHelperError("invalid_output", "Photos helper output is too large."));
return;
}
stdout.push(chunk);
});
child.stderr?.on("data", (chunk: Buffer) => {
const text = chunk.toString("utf8");
if (stderr.length < MAX_STDERR_BYTES) stderr += text;
const lines = (pendingLine + text).split("\n");
pendingLine = lines.pop() ?? "";
for (const line of lines) {
const progress = parseProgressLine(line.trim());
if (progress !== null) {
armIdle();
options.onProgress?.(progress);
}
}
});
child.on("error", () => {
finish(new PhotosHelperError("unavailable", "Photos helper could not be started."));
});
child.on("close", (code) => {
if (code === 0) finish(null, Buffer.concat(stdout).toString("utf8"));
else finish(parseHelperError(stderr + pendingLine, code));
});
});
}
const QUERY_TIMEOUT_MS = 2 * 60_000;
/** The system permission dialog waits on the person. */
const AUTHORIZE_TIMEOUT_MS = 10 * 60_000;
/** Large iCloud videos take long; a stalled download fails sooner through the idle limit. */
const EXPORT_TIMEOUT_MS = 2 * 60 * 60_000;
const EXPORT_IDLE_TIMEOUT_MS = 5 * 60_000;
export interface PhotosHelper {
authorize(prompt: boolean, signal?: AbortSignal): Promise<PhotosAuthorization>;
list(
request: { cursor: string | null; limit: number; since?: string | null },
signal?: AbortSignal,
): Promise<PhotosListPage>;
asset(localIdentifier: string, signal?: AbortSignal): Promise<PhotosAssetDetail>;
exportOriginal(
localIdentifier: string,
destination: string,
options?: { signal?: AbortSignal; onProgress?: (fraction: number) => void },
): Promise<PhotosExport>;
changes(token: string, signal?: AbortSignal): Promise<PhotosChanges>;
}
export function createPhotosHelper(binary: string, spawn?: SpawnHelper): PhotosHelper {
const run = (args: string[], options: RunOptions) =>
runPhotosHelper(binary, args, { ...options, spawn });
return {
async authorize(prompt, signal) {
const timeoutMs = prompt ? AUTHORIZE_TIMEOUT_MS : QUERY_TIMEOUT_MS;
return parseAuthorizeOutput(await run(authorizeArgs(prompt), { signal, timeoutMs }));
},
async list(request, signal) {
const output = await run(listArgs(request), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseListOutput(output, request.limit);
},
async asset(localIdentifier, signal) {
const output = await run(assetArgs(localIdentifier), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseAssetOutput(output);
},
async exportOriginal(localIdentifier, destination, options = {}) {
const output = await run(exportArgs(localIdentifier, destination), {
signal: options.signal,
onProgress: options.onProgress,
timeoutMs: EXPORT_TIMEOUT_MS,
idleTimeoutMs: EXPORT_IDLE_TIMEOUT_MS,
});
return parseExportOutput(output, destination);
},
async changes(token, signal) {
const output = await run(changesArgs(token), { signal, timeoutMs: QUERY_TIMEOUT_MS });
return parseChangesOutput(output);
},
};
}

View File

@ -0,0 +1,655 @@
import { createHash } from "node:crypto";
import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { ClientMediaItem, MediaAsset, MediaUpload } from "@rakazo/contracts";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { PhotosHelper } from "./media-photos-helper.js";
import type { MediaSyncApi, PhotosSyncState, PhotosSyncStore } from "./media-photos-sync.js";
import {
createChunkUploader,
createPhotosSyncFileStore,
parsePhotosSyncState,
runPool,
syncApplePhotos,
} from "./media-photos-sync.js";
const sha = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
function libraryItem(index: number): ClientMediaItem {
return {
externalId: `ASSET-${String(index).padStart(5, "0")}/L0/001`,
kind: index % 3 === 0 ? "video" : "photo",
capturedAt: new Date(Date.UTC(2024, 0, 1) + index * 60_000).toISOString(),
durationMs: index % 3 === 0 ? 5000 : null,
width: 1920,
height: 1080,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
fileName: `IMG_${index}.JPG`,
};
}
/** Pages a synthetic library the way the Swift helper does: opaque cursor = next index. */
function fakeHelper(options: { size: number; contents?: (id: string) => string }) {
const library = Array.from({ length: options.size }, (_, index) => libraryItem(index));
const exports: string[] = [];
let active = 0;
const stats = { maxActiveExports: 0 };
const helper: PhotosHelper & { library: ClientMediaItem[] } = {
library,
authorize: vi.fn(async () => "authorized" as const),
list: vi.fn(async ({ cursor, limit }) => {
const start = cursor === null ? 0 : Number(cursor);
const items = library.slice(start, start + limit);
const next = start + limit < library.length ? String(start + limit) : null;
return { items, nextCursor: next, changeToken: `token-${library.length}` };
}),
asset: vi.fn(),
changes: vi.fn(),
exportOriginal: vi.fn(async (id: string, destination: string, opts) => {
active += 1;
stats.maxActiveExports = Math.max(stats.maxActiveExports, active);
try {
await new Promise((resolve) => setTimeout(resolve, 5));
opts?.signal?.throwIfAborted();
const contents = options.contents?.(id) ?? `original bytes of ${id}`;
const file = path.join(destination, "IMG.MOV");
await writeFile(file, contents);
exports.push(destination);
return {
path: file,
fileName: "IMG.MOV",
sizeBytes: Buffer.byteLength(contents),
sha256: sha(contents),
};
} finally {
active -= 1;
}
}),
};
return { helper, exports, stats };
}
function memoryStore(initial?: PhotosSyncState) {
let state: PhotosSyncState = initial ?? { version: 1, deviceId: "device-1", targets: {} };
const writes: PhotosSyncState[] = [];
const store: PhotosSyncStore = {
read: async () => JSON.parse(JSON.stringify(state)) as PhotosSyncState,
write: async (next) => {
state = JSON.parse(JSON.stringify(next)) as PhotosSyncState;
writes.push(state);
},
};
return { store, writes, current: () => state };
}
function asset(id: string, externalId: string, ingestState: MediaAsset["ingestState"]) {
return {
id,
sourceId: "source-1",
externalId,
kind: "video",
storage: "remote",
capturedAt: null,
durationMs: null,
width: null,
height: null,
fps: null,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
ingestState,
hasProxy: false,
hasThumbnail: false,
qualityScore: null,
createdAt: "2024-01-01T00:00:00.000Z",
} as MediaAsset;
}
function fakeApi(
options: {
assets?: MediaAsset[];
chunkBytes?: number;
uploads?: Record<string, MediaUpload>;
} = {},
) {
const uploads = new Map(Object.entries(options.uploads ?? {}));
const pushes: {
items: number;
syncToken: string | null;
complete: boolean;
deleted?: number;
}[] = [];
const chunkBytes = options.chunkBytes ?? 4;
const api = {
media: {
sources: {
registerClient: vi.fn(async (input: { deviceId: string }) => ({
id: "source-1",
kind: "apple_photos" as const,
label: "Apple Photos",
status: "idle" as const,
assetCount: 0,
lastSyncedAt: null,
error: null,
createdAt: "2024-01-01T00:00:00.000Z",
deviceId: input.deviceId,
})),
pushItems: vi.fn(
async (input: {
items: unknown[];
deletedExternalIds?: string[];
syncToken: string | null;
complete: boolean;
}) => {
const deleted = input.deletedExternalIds?.length ?? 0;
expect(input.items.length > 0 || deleted > 0 || input.complete).toBe(true);
expect(input.items.length).toBeLessThanOrEqual(500);
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
...(deleted > 0 ? { deleted } : {}),
});
return { created: input.items.length, updated: 0, unchanged: 0, deleted };
},
),
},
assets: {
search: vi.fn(async (input: { cursor?: string | null }) => {
const all = options.assets ?? [];
// Two pages to exercise pagination.
const half = Math.ceil(all.length / 2);
return input.cursor === "p2"
? { items: all.slice(half), nextCursor: null }
: { items: all.slice(0, half), nextCursor: all.length > 1 ? "p2" : null };
}),
},
uploads: {
begin: vi.fn(async (input: { assetId: string; sizeBytes: number }) => {
const upload: MediaUpload = {
id: `upload-${input.assetId}`,
assetId: input.assetId,
sizeBytes: input.sizeBytes,
chunkBytes,
receivedChunks: [],
status: "open",
};
uploads.set(upload.id, upload);
return upload;
}),
status: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId);
if (!upload) throw Object.assign(new Error("Not found"), { code: "NOT_FOUND" });
return upload;
}),
complete: vi.fn(async (input: { uploadId: string }) => {
const upload = uploads.get(input.uploadId)!;
return { ...upload, status: "complete" as const };
}),
},
},
};
return { api: api as unknown as MediaSyncApi & typeof api, pushes };
}
let tempRoot: string;
beforeEach(async () => {
tempRoot = await mkdtemp(path.join(tmpdir(), "rakazo-photos-sync-"));
});
afterEach(async () => {
await rm(tempRoot, { recursive: true, force: true });
});
const baseOptions = () => ({
target: "https://rakazo.example.com#space_1",
label: "Apple Photos",
tempRoot,
signal: new AbortController().signal,
});
describe("Apple Photos metadata sync", () => {
it("registers once, pushes batches of at most 500 and completes the listing", async () => {
const { helper } = fakeHelper({ size: 1203 });
const { api, pushes } = fakeApi();
const { store, current } = memoryStore();
const chunks = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledWith(
{ kind: "apple_photos", label: "Apple Photos", deviceId: "device-1" },
expect.anything(),
);
expect(pushes).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 500, syncToken: null, complete: false },
{ items: 203, syncToken: "token-1203", complete: true },
]);
expect(result).toMatchObject({ indexed: 1203, created: 1203, uploaded: 0, fullListing: true });
expect(current().targets["https://rakazo.example.com#space_1"]).toEqual({
sourceId: "source-1",
listing: null,
changeToken: "token-1203",
uploads: {},
});
// The next run reads changes only and does not register again.
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "token-next",
changed: [libraryItem(1), libraryItem(2)],
deleted: [],
fullResyncRequired: false,
});
const second = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: chunks,
});
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(helper.changes).toHaveBeenCalledWith("token-1203", expect.anything());
expect(pushes.at(-1)).toEqual({ items: 2, syncToken: "token-next", complete: false });
expect(second.fullListing).toBe(false);
expect(current().targets["https://rakazo.example.com#space_1"]?.changeToken).toBe("token-next");
});
it("persists the cursor after every batch and resumes an interrupted listing", async () => {
const { helper } = fakeHelper({ size: 1100 });
const { api, pushes } = fakeApi();
const { store, writes, current } = memoryStore();
vi.mocked(api.media.sources.pushItems).mockImplementationOnce(async (input) => {
pushes.push({
items: input.items.length,
syncToken: input.syncToken,
complete: input.complete,
});
return { created: input.items.length, updated: 0, unchanged: 0 };
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(new Error("network down"));
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() }),
).rejects.toThrow("network down");
const interrupted = current().targets["https://rakazo.example.com#space_1"]!;
expect(interrupted.listing).toMatchObject({ cursor: "500", token: "token-1100" });
expect(interrupted.listing?.lastItem?.externalId).toBe(libraryItem(499).externalId);
expect(writes.length).toBeGreaterThan(1);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(vi.mocked(helper.list).mock.calls.at(-2)?.[0]).toMatchObject({ cursor: "500" });
expect(pushes.slice(1)).toEqual([
{ items: 500, syncToken: null, complete: false },
{ items: 100, syncToken: "token-1100", complete: true },
]);
expect(current().targets["https://rakazo.example.com#space_1"]).toMatchObject({
listing: null,
changeToken: "token-1100",
});
});
it("closes a resumed listing whose last page came back empty", async () => {
const { helper } = fakeHelper({ size: 0 });
const { api, pushes } = fakeApi();
const last = libraryItem(7);
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: { cursor: "500", token: "token-a", lastItem: last },
changeToken: null,
uploads: {},
},
},
});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(pushes).toEqual([{ items: 0, syncToken: "token-a", complete: true }]);
});
it("pushes device deletions without relisting", async () => {
const { helper } = fakeHelper({ size: 3 });
const { api, pushes } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "old",
uploads: {},
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "new",
changed: [],
deleted: ["gone/L0/001"],
fullResyncRequired: false,
});
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store,
uploadChunk: vi.fn(),
});
expect(result.fullListing).toBe(false);
expect(pushes).toEqual([{ items: 0, syncToken: "new", complete: false, deleted: 1 }]);
});
it("registers again when the server no longer knows the source", async () => {
const { helper } = fakeHelper({ size: 2 });
const { api } = fakeApi();
const { store } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "deleted-source",
listing: { cursor: null, token: null, lastItem: null },
changeToken: null,
uploads: {},
},
},
});
vi.mocked(api.media.sources.pushItems).mockRejectedValueOnce(
Object.assign(new Error("Not found"), { code: "NOT_FOUND" }),
);
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk: vi.fn() });
expect(api.media.sources.registerClient).toHaveBeenCalledTimes(1);
expect(vi.mocked(api.media.sources.pushItems).mock.calls.at(-1)?.[0].sourceId).toBe("source-1");
});
it("stops before anything is sent when Photos access is denied", async () => {
const { helper } = fakeHelper({ size: 2 });
vi.mocked(helper.authorize).mockResolvedValueOnce("denied");
const { api } = fakeApi();
await expect(
syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk: vi.fn(),
}),
).rejects.toMatchObject({ code: "not_authorized" });
expect(api.media.sources.registerClient).not.toHaveBeenCalled();
});
});
describe("Apple Photos original upload", () => {
const awaitingAssets = [
asset("a1", "ASSET-00001/L0/001", "awaiting_upload"),
asset("a2", "ASSET-00002/L0/001", "proxied"),
asset("a3", "ASSET-00003/L0/001", "awaiting_upload"),
asset("a4", "ASSET-00004/L0/001", "awaiting_upload"),
];
it("uploads every awaiting original in chunks with two at a time and cleans up", async () => {
const { helper, exports, stats } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: awaitingAssets, chunkBytes: 8 });
const sent = new Map<string, Buffer[]>();
const uploadChunk = vi.fn(async (uploadId: string, index: number, bytes: Uint8Array) => {
const parts = sent.get(uploadId) ?? [];
parts[index] = Buffer.from(bytes);
sent.set(uploadId, parts);
});
const progress = vi.fn();
const result = await syncApplePhotos({
...baseOptions(),
helper,
api,
store: memoryStore().store,
uploadChunk,
onProgress: progress,
});
expect(result.uploaded).toBe(3);
expect(
vi
.mocked(helper.exportOriginal)
.mock.calls.map((call) => call[0])
.sort(),
).toEqual(["ASSET-00001/L0/001", "ASSET-00003/L0/001", "ASSET-00004/L0/001"]);
expect(stats.maxActiveExports).toBe(2);
expect(Buffer.concat(sent.get("upload-a1")!).toString()).toBe(
"original bytes of ASSET-00001/L0/001",
);
expect(sent.get("upload-a1")).toHaveLength(Math.ceil(36 / 8));
expect(api.media.uploads.begin).toHaveBeenCalledWith(
expect.objectContaining({
assetId: "a1",
sizeBytes: 36,
sha256: sha("original bytes of ASSET-00001/L0/001"),
}),
expect.anything(),
);
expect(api.media.uploads.complete).toHaveBeenCalledTimes(3);
expect(progress).toHaveBeenLastCalledWith({
phase: "uploading",
indexed: 1,
uploadsDone: 3,
uploadsTotal: 3,
});
for (const dir of exports) await expect(readdir(dir)).rejects.toThrow();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("resumes an open upload and sends only the missing chunks", async () => {
const { helper } = fakeHelper({ size: 1 });
const contents = "original bytes of ASSET-00001/L0/001";
const { api } = fakeApi({
assets: [awaitingAssets[0]!],
uploads: {
"upload-old": {
id: "upload-old",
assetId: "a1",
sizeBytes: Buffer.byteLength(contents),
chunkBytes: 10,
receivedChunks: [0, 2],
status: "open",
},
},
});
const { store, current } = memoryStore({
version: 1,
deviceId: "device-1",
targets: {
"https://rakazo.example.com#space_1": {
sourceId: "source-1",
listing: null,
changeToken: "tok",
uploads: { a1: "upload-old" },
},
},
});
vi.mocked(helper.changes).mockResolvedValueOnce({
token: "tok",
changed: [],
deleted: [],
fullResyncRequired: false,
});
const uploadChunk = vi.fn(async () => {});
await syncApplePhotos({ ...baseOptions(), helper, api, store, uploadChunk });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(
uploadChunk.mock.calls.map((call) => [
call[0],
call[1],
Buffer.from(call[2] as Uint8Array).toString(),
]),
).toEqual([
["upload-old", 1, contents.slice(10, 20)],
["upload-old", 3, contents.slice(30)],
]);
expect(current().targets["https://rakazo.example.com#space_1"]?.uploads).toEqual({});
});
it("aborts on a checksum mismatch without starting an upload", async () => {
const { helper } = fakeHelper({ size: 1 });
vi.mocked(helper.exportOriginal).mockImplementationOnce(async (_id, destination) => {
const file = path.join(destination, "IMG.MOV");
await writeFile(file, "tampered");
return { path: file, fileName: "IMG.MOV", sizeBytes: 8, sha256: sha("original") };
});
const { api } = fakeApi({ assets: [awaitingAssets[0]!] });
const uploadChunk = vi.fn();
await expect(
syncApplePhotos({ ...baseOptions(), helper, api, store: memoryStore().store, uploadChunk }),
).rejects.toMatchObject({ code: "integrity" });
expect(api.media.uploads.begin).not.toHaveBeenCalled();
expect(uploadChunk).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
it("cancels a running upload", async () => {
const { helper } = fakeHelper({ size: 1 });
const { api } = fakeApi({ assets: [awaitingAssets[0]!], chunkBytes: 4 });
const controller = new AbortController();
const uploadChunk = vi.fn(
async (_id: string, index: number, _bytes: Uint8Array, signal: AbortSignal) => {
if (index === 1) controller.abort();
signal.throwIfAborted();
},
);
await expect(
syncApplePhotos({
...baseOptions(),
signal: controller.signal,
helper,
api,
store: memoryStore().store,
uploadChunk,
}),
).rejects.toMatchObject({ code: "cancelled" });
expect(uploadChunk).toHaveBeenCalledTimes(2);
expect(api.media.uploads.complete).not.toHaveBeenCalled();
await expect(readdir(tempRoot)).resolves.toEqual([]);
});
});
describe("Apple Photos sync state file", () => {
it("keeps a stable device id across launches", async () => {
const dir = await mkdtemp(path.join(tmpdir(), "rakazo-photos-state-"));
try {
const store = createPhotosSyncFileStore(dir, () => "device-a");
const first = await store.read();
expect(first.deviceId).toBe("device-a");
first.targets.x = { sourceId: "s", listing: null, changeToken: "t", uploads: {} };
await store.write(first);
const reopened = await createPhotosSyncFileStore(dir, () => "device-b").read();
expect(reopened).toEqual(first);
expect(await readFile(path.join(dir, "media-photos-sync.json"), "utf8")).toContain(
"device-a",
);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it("drops unreadable state and malformed targets", () => {
expect(parsePhotosSyncState("{")).toBeNull();
expect(
parsePhotosSyncState(JSON.stringify({ version: 2, deviceId: "d", targets: {} })),
).toBeNull();
expect(
parsePhotosSyncState(
JSON.stringify({
version: 1,
deviceId: "d",
targets: {
good: { sourceId: "s", listing: null, changeToken: null, uploads: {} },
bad: { sourceId: 4, listing: null, changeToken: null, uploads: {} },
badListing: { sourceId: "s", listing: { cursor: 3 }, changeToken: null, uploads: {} },
},
}),
)?.targets,
).toEqual({ good: { sourceId: "s", listing: null, changeToken: null, uploads: {} } });
});
});
describe("chunk uploader", () => {
it("PUTs raw bytes with the session and space", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 204 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: "space_1",
fetch,
});
const signal = new AbortController().signal;
await upload("up/1", 3, new Uint8Array([1, 2, 3]), signal);
expect(fetch).toHaveBeenCalledWith(
"https://rakazo.example.com/api/media/uploads/up%2F1/chunks/3",
expect.objectContaining({
method: "PUT",
credentials: "include",
redirect: "error",
signal,
headers: { "content-type": "application/octet-stream", "x-rakazo-space-id": "space_1" },
}),
);
});
it("rejects failed chunks and non-origin targets", async () => {
const fetch = vi.fn(async () => new Response(null, { status: 409 }));
const upload = createChunkUploader({
origin: "https://rakazo.example.com",
spaceId: null,
fetch,
});
await expect(
upload("u", 0, new Uint8Array(), new AbortController().signal),
).rejects.toMatchObject({
code: "upload_failed",
});
expect(() =>
createChunkUploader({ origin: "https://rakazo.example.com/rpc", spaceId: null, fetch }),
).toThrow();
expect(() => createChunkUploader({ origin: "file:///etc", spaceId: null, fetch })).toThrow();
});
});
describe("runPool", () => {
it("bounds concurrency and reports the first failure", async () => {
let active = 0;
let max = 0;
const seen: number[] = [];
await expect(
runPool([1, 2, 3, 4, 5], 2, new AbortController().signal, async (item, signal) => {
active += 1;
max = Math.max(max, active);
await new Promise((resolve) => setTimeout(resolve, 5));
active -= 1;
if (item === 2) throw new Error("boom");
signal.throwIfAborted();
seen.push(item);
}),
).rejects.toThrow("boom");
expect(max).toBe(2);
expect(seen).not.toContain(4);
});
});

View File

@ -0,0 +1,590 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { mkdtemp, open, rm, stat } from "node:fs/promises";
import path from "node:path";
import type { ClientMediaItem, MediaAsset, MediaUpload, mediaRpcShapes } from "@rakazo/contracts";
import type { PhotosHelper } from "./media-photos-helper.js";
import { PHOTOS_MAX_PAGE } from "./media-photos-helper.js";
import { readPrivateFile, writePrivateFile } from "./setup-store.js";
/*
* Apple Photos is a client-pushed source: the API may run on another machine, so the
* app never sends a local path. It pushes metadata in batches and uploads the originals
* the server asks for (`awaiting_upload`) in resumable raw-byte chunks.
*/
type Shapes = typeof mediaRpcShapes;
type Shape = { input: { _input: unknown }; output: { _output: unknown } };
/** One oRPC procedure as `ContractRouterClient<AppContract>` exposes it. */
type Procedure<S extends Shape> = (
input: S["input"]["_input"],
options?: { signal?: AbortSignal },
) => Promise<S["output"]["_output"]>;
/** The slice of the app RPC client the sync needs; `rpc` from a contract client satisfies it. */
export interface MediaSyncApi {
media: {
sources: {
registerClient: Procedure<Shapes["sources"]["registerClient"]>;
pushItems: Procedure<Shapes["sources"]["pushItems"]>;
};
assets: { search: Procedure<Shapes["assets"]["search"]> };
uploads: {
begin: Procedure<Shapes["uploads"]["begin"]>;
status: Procedure<Shapes["uploads"]["status"]>;
complete: Procedure<Shapes["uploads"]["complete"]>;
};
};
}
/** Sends one raw chunk to `PUT /api/media/uploads/:uploadId/chunks/:index`. */
export type UploadChunk = (
uploadId: string,
index: number,
bytes: Uint8Array<ArrayBuffer>,
signal: AbortSignal,
) => Promise<void>;
export type PhotosSyncErrorCode = "not_authorized" | "cancelled" | "integrity" | "upload_failed";
export class PhotosSyncError extends Error {
constructor(
readonly code: PhotosSyncErrorCode,
message: string,
) {
super(message);
this.name = "PhotosSyncError";
}
}
// ---------------------------------------------------------------------------
// Persistent state
// ---------------------------------------------------------------------------
export interface PhotosSyncTargetState {
sourceId: string;
/** A full listing in progress: where to resume, the token from its start and the last pushed item. */
listing: { cursor: string | null; token: string | null; lastItem: ClientMediaItem | null } | null;
/** Set after a complete listing; later syncs only read changes since it. */
changeToken: string | null;
/** Open uploads by asset id so an interrupted upload resumes instead of restarting. */
uploads: Record<string, string>;
}
export interface PhotosSyncState {
version: 1;
/** Stable per installation; the server keys the Apple Photos source by it. */
deviceId: string;
/** Keyed by server origin and space, since each has its own source. */
targets: Record<string, PhotosSyncTargetState>;
}
export interface PhotosSyncStore {
read(): Promise<PhotosSyncState>;
write(state: PhotosSyncState): Promise<void>;
}
export const PHOTOS_SYNC_FILE_NAME = "media-photos-sync.json";
const MAX_STATE_BYTES = 4 * 1024 * 1024;
/** Kept next to the desktop setup file, private to the user and written atomically. */
export function createPhotosSyncFileStore(
userDataDir: string,
newDeviceId: () => string = randomUUID,
): PhotosSyncStore {
const file = path.join(userDataDir, PHOTOS_SYNC_FILE_NAME);
return {
async read() {
const raw = await readPrivateFile(file, MAX_STATE_BYTES);
return (raw === null ? null : parsePhotosSyncState(raw)) ?? emptyState(newDeviceId());
},
write: (state) => writePrivateFile(file, `${JSON.stringify(state)}\n`),
};
}
function emptyState(deviceId: string): PhotosSyncState {
return { version: 1, deviceId, targets: {} };
}
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === "object" && value !== null && !Array.isArray(value);
const isNullableString = (value: unknown) => value === null || typeof value === "string";
/** Unreadable state starts over: the server deduplicates by device and external id. */
export function parsePhotosSyncState(raw: string): PhotosSyncState | null {
let value: unknown;
try {
value = JSON.parse(raw);
} catch {
return null;
}
if (
!isRecord(value) ||
value.version !== 1 ||
typeof value.deviceId !== "string" ||
value.deviceId === "" ||
value.deviceId.length > 200 ||
!isRecord(value.targets)
) {
return null;
}
const targets: Record<string, PhotosSyncTargetState> = {};
for (const [key, target] of Object.entries(value.targets)) {
if (
!isRecord(target) ||
typeof target.sourceId !== "string" ||
!isNullableString(target.changeToken) ||
!isRecord(target.uploads) ||
!Object.values(target.uploads).every((id) => typeof id === "string")
) {
continue;
}
const listing = target.listing;
const validListing =
listing === null ||
(isRecord(listing) &&
isNullableString(listing.cursor) &&
isNullableString(listing.token) &&
(listing.lastItem === null || isRecord(listing.lastItem)));
if (!validListing) continue;
targets[key] = target as unknown as PhotosSyncTargetState;
}
return { version: 1, deviceId: value.deviceId, targets };
}
// ---------------------------------------------------------------------------
// Chunk upload over HTTP
// ---------------------------------------------------------------------------
/**
* Chunk PUTs ride the same signed-in session as the app window: the injected `fetch`
* must carry that session's cookies (Electron `session.fetch` with credentials).
*/
export function createChunkUploader(options: {
origin: string;
spaceId: string | null;
fetch: (url: string, init: RequestInit) => Promise<Response>;
}): UploadChunk {
const origin = new URL(options.origin);
if (!["http:", "https:"].includes(origin.protocol) || origin.origin !== options.origin) {
throw new Error("Uploads need the server origin.");
}
return async (uploadId, index, bytes, signal) => {
if (!Number.isSafeInteger(index) || index < 0) throw new Error("Invalid chunk index.");
const headers: Record<string, string> = { "content-type": "application/octet-stream" };
if (options.spaceId) headers["x-rakazo-space-id"] = options.spaceId;
const response = await options.fetch(
`${origin.origin}/api/media/uploads/${encodeURIComponent(uploadId)}/chunks/${index}`,
{
method: "PUT",
headers,
body: bytes,
credentials: "include",
redirect: "error",
signal,
},
);
if (!response.ok) {
throw new PhotosSyncError(
"upload_failed",
`The server rejected an upload (${response.status}).`,
);
}
};
}
// ---------------------------------------------------------------------------
// Sync
// ---------------------------------------------------------------------------
export interface PhotosSyncProgress {
phase: "authorizing" | "indexing" | "uploading";
/** Items pushed in this run. */
indexed: number;
uploadsDone: number;
uploadsTotal: number;
}
export interface PhotosSyncResult {
indexed: number;
created: number;
updated: number;
unchanged: number;
uploaded: number;
/** True when this run listed the whole library instead of reading changes. */
fullListing: boolean;
}
export interface PhotosSyncOptions {
helper: PhotosHelper;
api: MediaSyncApi;
uploadChunk: UploadChunk;
store: PhotosSyncStore;
/** Server origin plus space, e.g. `https://rakazo.example.com#space_1`. */
target: string;
/** Source label shown in the library, e.g. the Mac's name or "Apple Photos". */
label: string;
/** Exports land in a fresh folder below this and are deleted after upload. */
tempRoot: string;
signal: AbortSignal;
onProgress?: (progress: PhotosSyncProgress) => void;
/** At most 500, the contract's batch limit. */
batchSize?: number;
concurrency?: number;
}
const MAX_SYNC_TOKEN_LENGTH = 2000;
export async function syncApplePhotos(options: PhotosSyncOptions): Promise<PhotosSyncResult> {
const { helper, api, signal } = options;
const batchSize = Math.min(Math.max(1, options.batchSize ?? PHOTOS_MAX_PAGE), PHOTOS_MAX_PAGE);
const result: PhotosSyncResult = {
indexed: 0,
created: 0,
updated: 0,
unchanged: 0,
uploaded: 0,
fullListing: false,
};
const progress = { uploadsDone: 0, uploadsTotal: 0 };
const report = (phase: PhotosSyncProgress["phase"]) =>
options.onProgress?.({ phase, indexed: result.indexed, ...progress });
const state = await options.store.read();
// Writes are serialized so concurrent upload workers never interleave partial state.
let saving = Promise.resolve();
const save = () => {
const snapshot = JSON.parse(JSON.stringify(state)) as PhotosSyncState;
saving = saving.then(() => options.store.write(snapshot));
return saving;
};
try {
report("authorizing");
const authorization = await helper.authorize(true, signal);
if (authorization !== "authorized" && authorization !== "limited") {
throw new PhotosSyncError("not_authorized", "Photos access was not granted.");
}
const ensureTarget = async (): Promise<PhotosSyncTargetState> => {
const existing = state.targets[options.target];
if (existing) return existing;
const source = await api.media.sources.registerClient(
{ kind: "apple_photos", label: options.label, deviceId: state.deviceId },
{ signal },
);
const created: PhotosSyncTargetState = {
sourceId: source.id,
listing: null,
changeToken: null,
uploads: {},
};
state.targets[options.target] = created;
await save();
return created;
};
const push = async (
target: PhotosSyncTargetState,
items: ClientMediaItem[],
syncToken: string | null,
complete: boolean,
deletedExternalIds: string[] = [],
) => {
const counts = await api.media.sources.pushItems(
{
sourceId: target.sourceId,
items,
deletedExternalIds,
syncToken:
syncToken !== null && syncToken.length <= MAX_SYNC_TOKEN_LENGTH ? syncToken : null,
complete,
},
{ signal },
);
result.indexed += items.length;
result.created += counts.created;
result.updated += counts.updated;
result.unchanged += counts.unchanged;
report("indexing");
};
/** Full listing, resumable page by page; the last push says `complete`. */
const listAll = async (target: PhotosSyncTargetState) => {
result.fullListing = true;
target.listing ??= { cursor: null, token: null, lastItem: null };
const listing = target.listing;
while (true) {
const page = await helper.list({ cursor: listing.cursor, limit: batchSize }, signal);
if (listing.cursor === null) listing.token = page.changeToken;
const done = page.nextCursor === null;
const items = page.items;
// An empty final page still closes the listing.
if (items.length > 0 || done) await push(target, items, done ? listing.token : null, done);
if (done) {
target.changeToken = listing.token;
target.listing = null;
await save();
return;
}
listing.cursor = page.nextCursor;
listing.lastItem = items.at(-1) ?? listing.lastItem;
await save();
}
};
const syncMetadata = async () => {
const target = await ensureTarget();
if (target.listing !== null || target.changeToken === null) {
await listAll(target);
return;
}
const changes = await helper.changes(target.changeToken, signal);
if (changes.fullResyncRequired || changes.token === null) {
target.listing = null;
await listAll(target);
return;
}
const batches = Math.max(
Math.ceil(changes.changed.length / batchSize),
Math.ceil(changes.deleted.length / DELETED_BATCH_SIZE),
);
for (let index = 0; index < batches; index++) {
const batch = changes.changed.slice(index * batchSize, (index + 1) * batchSize);
const deleted = changes.deleted.slice(
index * DELETED_BATCH_SIZE,
(index + 1) * DELETED_BATCH_SIZE,
);
const last = index === batches - 1;
await push(target, batch, last ? changes.token : null, false, deleted);
}
target.changeToken = changes.token;
await save();
};
report("indexing");
try {
await syncMetadata();
} catch (error) {
// The source was removed on the server: register again and list from scratch.
if (!isNotFound(error) || signal.aborted) throw error;
delete state.targets[options.target];
await save();
await syncMetadata();
}
const target = state.targets[options.target]!;
const awaiting = await awaitingUploads(api, target.sourceId, signal);
progress.uploadsTotal = awaiting.length;
// Forget uploads for assets the server no longer waits for.
const waiting = new Set(awaiting.map((asset) => asset.id));
for (const assetId of Object.keys(target.uploads)) {
if (!waiting.has(assetId)) delete target.uploads[assetId];
}
await save();
report("uploading");
await runPool(awaiting, options.concurrency ?? 2, signal, async (asset, workerSignal) => {
await uploadAsset({ ...options, signal: workerSignal }, target, asset, save);
result.uploaded += 1;
progress.uploadsDone += 1;
report("uploading");
});
return result;
} catch (error) {
if (signal.aborted) throw new PhotosSyncError("cancelled", "Sync cancelled.");
throw error;
} finally {
await saving.catch(() => undefined);
}
}
/**
* Binds the sync to the server and space the request names. `connect` builds the RPC
* client and chunk uploader on the signed-in session for that space.
*/
export function createPhotosSyncRunner(deps: {
helper: PhotosHelper;
store: PhotosSyncStore;
tempRoot: string;
label: string;
connect: (
spaceId: string | null,
) => Promise<{ origin: string; api: MediaSyncApi; uploadChunk: UploadChunk }>;
}) {
return async (request: {
spaceId: string | null;
signal: AbortSignal;
onProgress: (progress: PhotosSyncProgress) => void;
}): Promise<PhotosSyncResult> => {
const { origin, api, uploadChunk } = await deps.connect(request.spaceId);
return syncApplePhotos({
helper: deps.helper,
store: deps.store,
tempRoot: deps.tempRoot,
label: deps.label,
api,
uploadChunk,
target: `${origin}#${request.spaceId ?? "default"}`,
signal: request.signal,
onProgress: request.onProgress,
});
};
}
function isNotFound(error: unknown): boolean {
return isRecord(error) && error.code === "NOT_FOUND";
}
interface AwaitingAsset {
id: string;
externalId: string;
}
/** Assets of this source the server is waiting for, mapped back to Photos items. */
const DELETED_BATCH_SIZE = 5000;
export async function awaitingUploads(
api: MediaSyncApi,
sourceId: string,
signal: AbortSignal,
): Promise<AwaitingAsset[]> {
const awaiting: AwaitingAsset[] = [];
let cursor: string | null = null;
do {
const page: { items: MediaAsset[]; nextCursor: string | null } = await api.media.assets.search(
{ sourceId, ingestStates: ["awaiting_upload"], cursor, limit: PHOTOS_MAX_PAGE },
{ signal },
);
for (const asset of page.items) {
if (asset.ingestState !== "awaiting_upload" || asset.sourceId !== sourceId) continue;
awaiting.push({ id: asset.id, externalId: asset.externalId });
}
cursor = page.nextCursor;
} while (cursor !== null);
return awaiting;
}
async function uploadAsset(
options: PhotosSyncOptions,
target: PhotosSyncTargetState,
asset: AwaitingAsset,
save: () => Promise<void>,
): Promise<void> {
const { api, signal } = options;
const directory = await mkdtemp(path.join(options.tempRoot, "rakazo-photos-"));
try {
const exported = await options.helper.exportOriginal(asset.externalId, directory, { signal });
// The helper's hash is checked against the bytes on disk before anything is sent.
const local = await hashFile(exported.path, signal);
if (local.sha256 !== exported.sha256 || local.sizeBytes !== exported.sizeBytes) {
throw new PhotosSyncError("integrity", "The exported file does not match its checksum.");
}
let upload: MediaUpload | null = null;
const previous = target.uploads[asset.id];
if (previous) {
const status = await api.media.uploads
.status({ uploadId: previous }, { signal })
.catch((error: unknown) => {
if (isNotFound(error)) return null;
throw error;
});
if (status?.status === "complete") {
delete target.uploads[asset.id];
await save();
return;
}
if (status?.status === "open" && status.sizeBytes === local.sizeBytes) upload = status;
}
if (upload === null) {
upload = await api.media.uploads.begin(
{
assetId: asset.id,
sizeBytes: local.sizeBytes,
sha256: local.sha256,
fileName: exported.fileName.slice(0, 255),
},
{ signal },
);
target.uploads[asset.id] = upload.id;
await save();
}
const received = new Set(upload.receivedChunks);
const chunkCount = Math.ceil(local.sizeBytes / upload.chunkBytes);
const file = await open(exported.path, "r");
try {
for (let index = 0; index < chunkCount; index += 1) {
if (received.has(index)) continue;
signal.throwIfAborted();
const offset = index * upload.chunkBytes;
const length = Math.min(upload.chunkBytes, local.sizeBytes - offset);
const bytes = Buffer.alloc(length);
const { bytesRead } = await file.read(bytes, 0, length, offset);
if (bytesRead !== length) {
throw new PhotosSyncError("integrity", "The exported file changed during upload.");
}
await options.uploadChunk(upload.id, index, bytes, signal);
}
} finally {
await file.close();
}
const completed = await api.media.uploads.complete({ uploadId: upload.id }, { signal });
if (completed.status !== "complete") {
throw new PhotosSyncError("upload_failed", "The server did not accept the upload.");
}
delete target.uploads[asset.id];
await save();
} finally {
await rm(directory, { recursive: true, force: true });
}
}
export async function hashFile(
file: string,
signal?: AbortSignal,
): Promise<{ sha256: string; sizeBytes: number }> {
const hash = createHash("sha256");
for await (const chunk of createReadStream(file, { signal })) hash.update(chunk as Buffer);
return { sha256: hash.digest("hex"), sizeBytes: (await stat(file)).size };
}
/**
* Runs `work` with at most `limit` items in flight. The first failure aborts the
* others and is the error reported.
*/
export async function runPool<T>(
items: T[],
limit: number,
signal: AbortSignal,
work: (item: T, signal: AbortSignal) => Promise<void>,
): Promise<void> {
const controller = new AbortController();
const forward = () => controller.abort(signal.reason);
signal.addEventListener("abort", forward, { once: true });
if (signal.aborted) forward();
let next = 0;
let failed = false;
let failure: unknown;
const worker = async () => {
while (!failed && !controller.signal.aborted && next < items.length) {
const item = items[next++]!;
try {
await work(item, controller.signal);
} catch (error) {
if (!failed) {
failed = true;
failure = error;
controller.abort(error);
}
}
}
};
try {
await Promise.all(Array.from({ length: Math.min(Math.max(1, limit), items.length) }, worker));
} finally {
signal.removeEventListener("abort", forward);
}
if (failed) throw failure;
signal.throwIfAborted();
}

View File

@ -35,6 +35,7 @@
"expo-image-picker": "57.0.20",
"expo-linking": "57.0.11",
"expo-localization": "57.0.2",
"expo-media-library": "57.0.5",
"expo-modules-core": "57.0.20",
"expo-notifications": "57.0.21",
"expo-router": "57.0.24",
@ -44,6 +45,7 @@
"expo-status-bar": "57.0.1",
"expo-symbols": "57.0.3",
"expo-updates": "57.0.24",
"expo-video": "57.0.5",
"react": "19.2.3",
"react-dom": "19.2.3",
"react-native": "0.86.3",

View File

@ -16,11 +16,12 @@ test.use({ viewport: { width: 1280, height: 1000 } });
/**
* Texts in `locator` below WCAG AA against the surface they are painted on. With `inside`, every
* visible text and typed value within it counts, except in disabled controls, which WCAG exempts.
* visible text and typed value within it counts, except in disabled controls, which WCAG exempts,
* and in elements matching `skip`.
*/
function unreadableTexts(locator: Locator, inside: boolean) {
function unreadableTexts(locator: Locator, inside: boolean, skip: string | null) {
return locator.evaluate(
(root, { inside, min }) => {
(root, { inside, skip, min }) => {
const context = document
.createElement("canvas")
.getContext("2d", { willReadFrequently: true });
@ -81,7 +82,11 @@ function unreadableTexts(locator: Locator, inside: boolean) {
};
const elements = inside
? [root, ...root.querySelectorAll("*")].filter(
(element) => ownText(element) && shown(element) && !element.closest(":disabled"),
(element) =>
ownText(element) &&
shown(element) &&
!element.closest(":disabled") &&
!(skip && element.closest(skip)),
)
: [root];
return elements.flatMap((element) => {
@ -90,14 +95,18 @@ function unreadableTexts(locator: Locator, inside: boolean) {
return ratio < min ? [`${ratio.toFixed(2)}:1 ${text.slice(0, 48)}`] : [];
});
},
{ inside, min: MIN_CONTRAST },
{ inside, skip, min: MIN_CONTRAST },
);
}
async function expectReadable(locator: Locator, what: string, { inside = false } = {}) {
async function expectReadable(
locator: Locator,
what: string,
{ inside = false, skip = null }: { inside?: boolean; skip?: string | null } = {},
) {
await expect(locator).toBeVisible();
await expect
.poll(() => unreadableTexts(locator, inside), { message: `${what} is readable` })
.poll(() => unreadableTexts(locator, inside, skip), { message: `${what} is readable` })
.toEqual([]);
}
@ -213,6 +222,9 @@ test("shell surfaces stay readable in light and dark themes", async ({ page }, t
await composer.fill("Draft a note about owls");
const sidebar = page.getByTestId("bots-sidebar");
const sidePanel = page.getByTestId("side-panel");
await expect(sidebar.locator("[data-main-conversation]")).toBeVisible();
await expect(sidebar.getByRole("button", { name: "Researcher, 1:1 conversation" })).toBeVisible();
await expect(page.getByTestId("trading-mode-status")).toBeVisible();
await inEachTheme(page, async (theme) => {
await captureScreenshot(page, testInfo, `shell-contrast-${theme}`);
@ -220,27 +232,9 @@ test("shell surfaces stay readable in light and dark themes", async ({ page }, t
await expectReadable(page.getByTestId("composer-bar"), `${theme}: the composer`, {
inside: true,
});
await expectReadable(
sidebar.getByRole("button", { name: "Heute", exact: true }),
`${theme}: Heute`,
{ inside: true },
);
await expectReadable(
sidebar.locator("[data-main-conversation]"),
`${theme}: the main conversation row`,
{ inside: true },
);
await expectReadable(
sidebar.getByRole("button", { name: "Researcher, 1:1 conversation" }),
`${theme}: the selected bot row`,
{ inside: true },
);
await expectReadable(page.getByTestId("trading-mode-status"), `${theme}: trading status`, {
inside: true,
});
await expectReadable(page.getByTestId("user-menu-trigger"), `${theme}: the user menu`, {
inside: true,
});
// The header carries the logo, which WCAG exempts; its create button is checked on its own.
await expectReadable(sidebar, `${theme}: the sidebar`, { inside: true, skip: ".app-drag" });
await expectReadable(page.getByTestId("create-menu-trigger"), `${theme}: the create button`);
});
await openNewBot(page);
@ -259,4 +253,12 @@ test("shell surfaces stay readable in light and dark themes", async ({ page }, t
await captureScreenshot(page, testInfo, `shell-contrast-${theme}-settings`);
await expectReadable(sidePanel, `${theme}: the bot settings form`, { inside: true });
});
await sidebar.getByRole("button", { name: "Heute", exact: true }).click();
const heute = page.getByTestId("heute-brief");
await expect(heute.getByTestId("heute-mission-control-link")).toBeVisible();
await inEachTheme(page, async (theme) => {
await captureScreenshot(page, testInfo, `shell-contrast-${theme}-heute`);
await expectReadable(heute, `${theme}: the Heute brief`, { inside: true });
});
});

View File

@ -3,6 +3,7 @@ import {
captureScreenshot,
completeOnboarding,
openNewSpace,
routePatchRpc,
rpc,
signup,
unterhaltungenOnly,
@ -45,6 +46,30 @@ test("Projekte + opens new project dialog instead of group form", async ({ page
await expect(page.getByTestId("project-start-main")).toHaveText(/Starten/);
});
for (const owner of [true, false]) {
const role = owner ? "owner" : "member";
const title = owner
? "new project dialog offers a host folder to the deployment owner"
: "new project dialog offers no host folder to a member";
test(title, async ({ page }, testInfo) => {
await signup(page, `proj-folder-${role}-${Date.now()}@rakazo.test`, "password12", "Folder");
await completeOnboarding(page);
// Whoever signs up first owns the deployment, which depends on spec order.
await routePatchRpc(page, "bootstrap", (json) => ({
...json,
me: { ...(json.me as Record<string, unknown>), isDeploymentOwner: owner },
}));
await page.goto("/app");
await page.waitForURL(/\/app\/[^/]+$/);
await page.locator(".sidebar-projekte").getByTestId("create-project-trigger").click();
const dialog = page.getByRole("dialog", { name: "New project" });
await expect(dialog.getByLabel("Name")).toBeVisible();
await expect(dialog.getByLabel("Project folder")).toHaveCount(owner ? 1 : 0);
await captureScreenshot(page, testInfo, `new-project-${role}`);
});
}
test("spaces stay invisible by default and chat creation requires approval", async ({
page,
}, testInfo) => {

View File

@ -24,6 +24,7 @@
"@rakazo/core": "workspace:*",
"@rakazo/ui-tokens": "workspace:*",
"@rakazo/ui-web": "workspace:*",
"@tanstack/react-virtual": "3.14.13",
"better-auth": "1.7.4",
"lucide-react": "1.44.0",
"react": "19.2.3",

View File

@ -10,16 +10,16 @@ const TRUST_ITEMS = [
export const DataTrustPanel = memo(function DataTrustPanel() {
return (
<div className="hidden xl:flex flex-col items-start gap-1.5 rounded-xl border border-border/40 bg-muted/30 px-3 py-2.5">
<div className="flex items-center gap-1.5 text-xs font-medium text-emerald-600 dark:text-emerald-400">
<CheckCircle2 className="size-3.5" />
<div className="flex items-center gap-1.5 text-xs font-medium text-sidebar-foreground">
<CheckCircle2 className="size-3.5 text-success" />
<span>Datenschutz aktiv</span>
</div>
<div className="flex flex-col gap-1">
{TRUST_ITEMS.map(({ icon: Icon, label, desc }) => (
<div key={label} className="flex items-center gap-1.5">
<Icon className="size-3 text-muted-foreground/60" />
<span className="text-[11px] leading-tight text-muted-foreground">{label}</span>
<span className="text-[10px] leading-tight text-muted-foreground/50">· {desc}</span>
<div key={label} className="flex items-center gap-1.5 rk-sidebar-muted">
<Icon className="size-3" />
<span className="text-[11px] leading-tight">{label}</span>
<span className="text-[10px] leading-tight">· {desc}</span>
</div>
))}
</div>

View File

@ -45,7 +45,7 @@ export function TradingModeIndicator({
<TradingModeLabel label={presentation.label} />
</span>
{presentation.subtitle ? (
<span className="block truncate text-ui-body-sm text-muted-foreground/70">
<span className="block truncate text-ui-body-sm rk-sidebar-muted">
<TradingModeSubtitleText subtitle={presentation.subtitle} />
</span>
) : null}

View File

@ -178,7 +178,7 @@ export function HeuteBrief({
href={MISSION_CONTROL_URL}
target="_blank"
rel="noreferrer"
className="inline-flex items-center gap-1.5 rounded-lg px-2 py-1 text-ui-label text-muted-foreground/70 transition-colors hover:text-muted-foreground"
className="inline-flex items-center gap-1.5 rounded-lg px-2 py-1 text-ui-label text-muted-foreground transition-colors hover:text-foreground"
data-testid="heute-mission-control-link"
>
<Trans>Mission Control</Trans>
@ -235,7 +235,7 @@ function EmptyDay({ onStartChat, hasBots }: { onStartChat: () => void; hasBots:
function ActionableChip({ label }: { label: string }) {
return (
<span className="shrink-0 rounded-full border border-warning/30 bg-warning/10 px-2 py-0.5 text-ui-label font-medium text-warning">
<span className="shrink-0 rounded-full border border-warning/30 bg-warning/10 px-2 py-0.5 text-ui-label font-medium text-foreground dark:text-warning">
{label}
</span>
);

View File

@ -3456,7 +3456,7 @@ export function ShellPage() {
className={`app-no-drag flex size-control-sm items-center justify-center rounded-full ${
activityMode
? "bg-primary text-primary-foreground"
: "text-muted-foreground/70 hover:text-foreground/75"
: "rk-sidebar-muted hover:text-foreground/75"
}`}
>
<Bell
@ -3469,7 +3469,7 @@ export function ShellPage() {
<Popover open={createMenuOpen} onOpenChange={setCreateMenuOpen}>
<PopoverTrigger
className="app-no-drag text-[21px] text-muted-foreground/70 hover:text-foreground/75"
className="app-no-drag text-[21px] rk-sidebar-muted hover:text-foreground/75"
title={t`Create`}
data-testid="create-menu-trigger"
>
@ -3592,7 +3592,7 @@ export function ShellPage() {
type="button"
aria-label={t`Neue Unterhaltung`}
data-testid="conversation-picker-trigger"
className="text-muted-foreground/60 hover:text-foreground/90 transition-colors"
className="rk-sidebar-muted hover:text-foreground/90 transition-colors"
>
<Plus strokeWidth={2.5} className="size-icon-md" />
</PopoverTrigger>
@ -3633,7 +3633,7 @@ export function ShellPage() {
</div>
) : null}
{conversationSidebarGroups.length === 0 && !mainConversationEntry ? (
<div className="p-4 text-center text-ui-label text-muted-foreground">
<div className="p-4 text-center text-ui-label rk-sidebar-muted">
<p>{t`Noch keine Unterhaltung.`}</p>
<Button
variant="outline"
@ -3657,7 +3657,7 @@ export function ShellPage() {
<div className="flex items-center pt-3 pb-0.5">
<button
type="button"
className="flex min-w-0 flex-1 items-center justify-between gap-2 rounded-lg px-2.5 py-1 text-ui-label font-semibold tracking-wider uppercase text-muted-foreground/60 hover:text-foreground focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-[-2px] focus-visible:outline-ring"
className="flex min-w-0 flex-1 items-center justify-between gap-2 rounded-lg px-2.5 py-1 text-ui-label font-semibold tracking-wider uppercase text-foreground/65 dark:text-muted-foreground/80 hover:text-foreground focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-[-2px] focus-visible:outline-ring"
onClick={() => {
if (group.emptySpaceId) {
openSpaceChat(group.emptySpaceId, "/onboarding");
@ -3841,7 +3841,7 @@ export function ShellPage() {
{item.chat.name}
</span>
{needsYouChipLabel(item.chat.status) ? (
<span className="shrink-0 rounded-full border border-warning/30 bg-warning/10 px-2 py-0.5 text-ui-label font-medium text-warning">
<span className="shrink-0 rounded-full border border-warning/30 bg-warning/10 px-2 py-0.5 text-ui-label font-medium text-foreground dark:text-warning">
{needsYouChipLabel(item.chat.status)}
</span>
) : item.chat.title ? (
@ -3887,7 +3887,7 @@ export function ShellPage() {
[group.key]: !prev[group.key],
}))
}
className="mt-1 w-full flex items-center justify-center gap-1.5 rounded-lg px-2 py-1.5 text-ui-label font-medium text-muted-foreground hover:bg-sidebar-accent hover:text-foreground transition-colors"
className="mt-1 w-full flex items-center justify-center gap-1.5 rounded-lg px-2 py-1.5 text-ui-label font-medium rk-sidebar-muted hover:bg-sidebar-accent hover:text-foreground transition-colors"
>
{expandedTeamGroups[group.key]
? "Weniger anzeigen"
@ -3902,7 +3902,7 @@ export function ShellPage() {
<div className="flex items-center justify-between pt-4 pb-1 mt-3 border-t border-sidebar-border px-2.5">
<div>
<div className="rk-shell-section-label">{t`Projekte`}</div>
<p className="mt-0.5 px-0.5 text-ui-caption text-muted-foreground/70">
<p className="mt-0.5 px-0.5 text-ui-caption rk-sidebar-muted">
{t`Threads by topic · folders`}
</p>
</div>
@ -3912,14 +3912,14 @@ export function ShellPage() {
aria-label={t`Create a project`}
data-testid="create-project-trigger"
onClick={() => openNewSpaceDialog("project")}
className="text-muted-foreground/60 hover:text-foreground/90 transition-colors"
className="rk-sidebar-muted hover:text-foreground/90 transition-colors"
>
<Plus strokeWidth={2.5} className="size-icon-md" />
</button>
</div>
</div>
{projectSidebarGroups.length === 0 ? (
<div className="p-4 text-center text-ui-label text-muted-foreground">
<div className="p-4 text-center text-ui-label rk-sidebar-muted">
<p>{t`No projects yet.`}</p>
<Button
variant="outline"
@ -3957,7 +3957,7 @@ export function ShellPage() {
</ProjectDragGrip>
<button
type="button"
className="flex min-w-0 flex-1 items-center justify-between gap-2 rounded-lg pl-8 py-1 text-ui-label font-semibold tracking-wider uppercase text-muted-foreground/60 hover:text-foreground focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-[-2px] focus-visible:outline-ring"
className="flex min-w-0 flex-1 items-center justify-between gap-2 rounded-lg pl-8 py-1 text-ui-label font-semibold tracking-wider uppercase text-foreground/65 dark:text-muted-foreground/80 hover:text-foreground focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-[-2px] focus-visible:outline-ring"
onClick={() => {
if (group.emptySpaceId) {
openSpaceChat(group.emptySpaceId, "/app");
@ -4153,7 +4153,7 @@ export function ShellPage() {
type="button"
aria-expanded={archivedOpen}
onClick={() => setArchivedOpen((open) => !open)}
className="flex w-full items-center justify-between rounded-lg px-2.5 py-2 text-ui-body text-muted-foreground hover:bg-sidebar-accent"
className="flex w-full items-center justify-between rounded-lg px-2.5 py-2 text-ui-body rk-sidebar-muted hover:bg-sidebar-accent"
>
<span>
<Trans>Archived</Trans>
@ -5482,6 +5482,7 @@ export function ShellPage() {
{newSpaceOpen ? (
<NewSpaceDialog
variant={newSpaceDialogVariant}
canLinkFolder={bootstrapMe?.isDeploymentOwner === true}
onCancel={() => setNewSpaceOpen(false)}
onConfirm={async ({ name, workspacePath }) => {
const space = await rpc.spaces.create({

View File

@ -38,10 +38,13 @@ export type NewSpaceConfirmInput = {
export function NewSpaceDialog({
variant = "space",
canLinkFolder,
onCancel,
onConfirm,
}: {
variant?: "space" | "project";
/** Folders live on the host, which only the deployment owner may link. */
canLinkFolder: boolean;
onCancel: () => void;
onConfirm: (input: NewSpaceConfirmInput) => Promise<void>;
}) {
@ -53,6 +56,7 @@ export function NewSpaceDialog({
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const isProject = variant === "project";
const showFolder = isProject && canLinkFolder;
const desktop = desktopBridge();
const trimmedWorkspacePath = workspacePath.trim();
@ -63,7 +67,7 @@ export function NewSpaceDialog({
setError(null);
void onConfirm({
name: trimmed,
...(isProject && trimmedWorkspacePath ? { workspacePath: trimmedWorkspacePath } : {}),
...(showFolder && trimmedWorkspacePath ? { workspacePath: trimmedWorkspacePath } : {}),
}).catch((reason: unknown) => {
setError(
reason instanceof Error
@ -104,7 +108,7 @@ export function NewSpaceDialog({
className="mt-2"
/>
</label>
{isProject ? (
{showFolder ? (
<div className="space-y-2">
<label htmlFor={folderId} className="block text-ui-body text-foreground/75">
<Trans>Project folder</Trans>

View File

@ -11,9 +11,7 @@ export function rosterNameClass(unread: boolean): string {
return unread ? "font-bold text-sidebar-foreground" : "font-semibold text-sidebar-foreground";
}
// Muted text measures about 4.4:1 on the light sidebar; a dimmed foreground stays above 4.5:1.
export const rosterTimeClass =
"text-ui-body-sm text-sidebar-foreground/65 tabular-nums font-medium";
export const rosterTimeClass = "text-ui-body-sm rk-sidebar-muted tabular-nums font-medium";
export const rosterUnreadDotClass = "inline-block h-2 w-2 rounded-full bg-sidebar-foreground";

View File

@ -73,7 +73,7 @@ body,
/* Shell layout utilities — token-backed, use instead of one-off pixel classes */
@layer components {
.rk-shell-section-label {
@apply text-ui-label font-bold tracking-wider uppercase text-muted-foreground/80;
@apply text-ui-label font-bold tracking-wider uppercase text-foreground/65 dark:text-muted-foreground/80;
}
.rk-shell-icon-btn {
@ -84,6 +84,12 @@ body,
@apply text-ui-body-sm text-muted-foreground;
}
/* Secondary sidebar text and icons. Muted text measures 4.4:1 on the light sidebar;
a dimmed foreground stays above 4.5:1. */
.rk-sidebar-muted {
@apply text-sidebar-foreground/65;
}
.rk-shell-row-title {
@apply text-ui-body font-medium text-foreground;
}

View File

@ -22,6 +22,12 @@ function handlers(): BackgroundJobHandlers {
"messaging.deliver": vi.fn(async () => undefined),
"cloud_agent.poll": vi.fn(async () => undefined),
"profile.consolidate": vi.fn(async () => undefined),
"media.sync": vi.fn(async () => undefined),
"media.proxy": vi.fn(async () => undefined),
"media.analyze": vi.fn(async () => undefined),
"media.cluster": vi.fn(async () => undefined),
"media.render": vi.fn(async () => undefined),
"media.publish": vi.fn(async () => undefined),
};
}

View File

@ -1,3 +1,5 @@
import type { MediaAnalysisKind } from "@rakazo/contracts";
import { MediaAnalysisKindSchema } from "@rakazo/contracts";
import { z } from "zod";
import type {
BackgroundJob,
@ -23,6 +25,15 @@ const payloadSchemas = {
"messaging.deliver": z.object({ runId: z.string().min(1).optional() }),
"cloud_agent.poll": z.object({ agentId: z.string().min(1) }),
"profile.consolidate": z.object({ userId: z.string().min(1) }),
"media.sync": z.object({ sourceId: z.string().min(1) }),
"media.proxy": z.object({ assetId: z.string().min(1) }),
"media.analyze": z.object({
assetId: z.string().min(1),
kinds: z.array(MediaAnalysisKindSchema).optional(),
}),
"media.cluster": z.object({ spaceId: z.string().min(1) }),
"media.render": z.object({ renderId: z.string().min(1) }),
"media.publish": z.object({ publicationId: z.string().min(1) }),
} satisfies { [Name in BackgroundJobName]: z.ZodType<BackgroundJobPayloads[Name]> };
export function parseBackgroundJob(name: string, payload: unknown): BackgroundJob {
@ -160,3 +171,41 @@ export function profileConsolidateJob(userId: string): BackgroundJob {
replaceKey: profileConsolidateJobKey(userId),
};
}
export function mediaSyncJob(sourceId: string): BackgroundJob {
return { name: "media.sync", payload: { sourceId }, replaceKey: `media.sync:${sourceId}` };
}
export function mediaProxyJob(assetId: string): BackgroundJob {
return { name: "media.proxy", payload: { assetId }, replaceKey: `media.proxy:${assetId}` };
}
export function mediaAnalyzeJob(assetId: string, kinds?: MediaAnalysisKind[]): BackgroundJob {
return {
name: "media.analyze",
payload: kinds ? { assetId, kinds } : { assetId },
replaceKey: `media.analyze:${assetId}`,
};
}
export function mediaClusterJob(spaceId: string, availableAt?: Date): BackgroundJob {
return {
name: "media.cluster",
payload: { spaceId },
replaceKey: `media.cluster:${spaceId}`,
...(availableAt ? { availableAt } : {}),
};
}
export function mediaRenderJob(renderId: string): BackgroundJob {
return { name: "media.render", payload: { renderId }, replaceKey: `media.render:${renderId}` };
}
export function mediaPublishJob(publicationId: string, availableAt?: Date): BackgroundJob {
return {
name: "media.publish",
payload: { publicationId },
replaceKey: `media.publish:${publicationId}`,
...(availableAt ? { availableAt } : {}),
};
}

View File

@ -1,5 +1,6 @@
export * from "./background-jobs.js";
export * from "./cloud-agents.js";
export * from "./interfaces.js";
export * from "./media.js";
export * from "./registry.js";
export * from "./types.js";

View File

@ -0,0 +1,325 @@
import type {
AnonymizeMode,
BoxKeyframe,
BrandKit,
ContentCategory,
FfprobeReport,
MediaAssetKind,
MediaCapabilities,
MediaInferenceOperation,
MediaMarkPrompt,
MediaSourceKind,
MotionTemplateInfo,
MotionTemplateRef,
NormalizedBox,
NormalizedPoint,
PlatformProfile,
RenderPlan,
} from "@rakazo/contracts";
/*
* Media Studio provider boundaries. Every implementation (local sidecar,
* emulator, optional cloud adapters) satisfies the same contracts, which
* media-conformance tests exercise offline.
*
* Paths are absolute host paths inside the configured media root. Providers
* must reject any path outside that root.
*/
// ---------------------------------------------------------------------------
// Sources
// ---------------------------------------------------------------------------
export interface MediaSourceItem {
externalId: string;
kind: MediaAssetKind;
capturedAt: Date | null;
durationMs: number | null;
width: number | null;
height: number | null;
fps: number | null;
gpsLat: number | null;
gpsLon: number | null;
deviceModel: string | null;
albumNames: string[];
isFavorite: boolean;
fileName: string | null;
sizeBytes: number | null;
/** Cheap change detector (mtime + size or provider revision); not a content hash. */
revision: string;
}
export interface MediaSourceListRequest {
config: Record<string, unknown>;
cursor: string | null;
limit: number;
signal?: AbortSignal;
}
export interface MediaSourceListResult {
items: MediaSourceItem[];
nextCursor: string | null;
}
/** Server-side sources the worker can enumerate and read directly. */
export interface MediaSourceProvider {
readonly kind: Extract<MediaSourceKind, "folder" | "emulator">;
list(request: MediaSourceListRequest): Promise<MediaSourceListResult>;
/** Absolute readable path of the original file. Throws when it escaped the source root. */
resolve(config: Record<string, unknown>, externalId: string): Promise<string>;
}
// ---------------------------------------------------------------------------
// Inference
// ---------------------------------------------------------------------------
export class UnsupportedMediaOperationError extends Error {
constructor(
readonly operation: MediaInferenceOperation,
readonly reason: string,
) {
super(`${operation} is not available: ${reason}`);
this.name = "UnsupportedMediaOperationError";
}
}
export interface MediaTimeSpan {
startMs: number;
endMs: number;
}
export interface DetectScenesResult {
shots: Array<MediaTimeSpan & { score: number }>;
scenes: MediaTimeSpan[];
}
export interface TranscriptWord {
startMs: number;
endMs: number;
text: string;
confidence: number;
}
export interface TranscribeResult {
language: string | null;
words: TranscriptWord[];
}
export interface FaceObservation {
box: NormalizedBox;
confidence: number;
/** Five-point landmarks: eyes, nose, mouth corners. */
landmarks?: NormalizedPoint[];
}
export interface DetectFacesResult {
frames: Array<{ tMs: number; faces: FaceObservation[] }>;
}
export interface FaceTrackResult {
trackId: string;
keyframes: Array<BoxKeyframe & { confidence: number }>;
/** Identity embedding for local person clustering. Never leaves the host. */
embedding: number[] | null;
}
export interface TrackFacesResult {
sampleFps: number;
tracks: FaceTrackResult[];
}
export interface FaceLandmarksResult {
frames: Array<{
tMs: number;
faces: Array<{ box: NormalizedBox; landmarks: NormalizedPoint[] }>;
}>;
}
export interface EmbedFramesResult {
model: string;
dim: number;
vectors: number[][];
}
export interface ClassifyContentResult {
model: string;
frames: Array<{ tMs: number; scores: Partial<Record<ContentCategory, number>> }>;
}
export interface SegmentPromptedResult {
/** Directory of per-frame PNG masks named by timestamp in ms. */
maskDir: string;
keyframes: Array<BoxKeyframe & { confidence: number }>;
score: number;
}
export interface InpaintQuality {
/** 1 = no flicker inside the mask region between neighbouring frames. */
temporalStability: number;
/** 1 = no visible seams at the mask edge. */
edgeQuality: number;
score: number;
}
export interface InpaintVideoResult {
outPath: string;
quality: InpaintQuality;
}
export interface AudioFeaturesResult {
integratedLufs: number | null;
truePeakDb: number | null;
tempoBpm: number | null;
beatsMs: number[];
energy: Array<{ tMs: number; value: number }>;
silences: MediaTimeSpan[];
}
export interface SaliencyResult {
points: Array<{ tMs: number; cx: number; cy: number; strength: number }>;
}
export interface RedactionInstruction {
keyframes: BoxKeyframe[];
mode: Exclude<AnonymizeMode, "none" | "reframe_out" | "synthetic_identity"> | "cover";
paddingPct: number;
featherPct: number;
/** Solid and cover fill. */
color?: string;
/** Landmarks per keyframe for partial modes. */
landmarks?: Array<{ tMs: number; points: NormalizedPoint[] }>;
/** Mask directory for region marks instead of boxes. */
maskDir?: string;
}
interface MediaPathRequest {
path: string;
signal?: AbortSignal;
}
/**
* Local-first ML operations. Callers check `capabilities()` and fall back
* (blur, cover, reframe) when an operation reports unavailable; providers
* throw UnsupportedMediaOperationError instead of guessing.
*/
export interface MediaInferenceProvider {
readonly id: string;
capabilities(signal?: AbortSignal): Promise<MediaCapabilities>;
detectScenes(request: MediaPathRequest): Promise<DetectScenesResult>;
transcribe(request: MediaPathRequest & { language?: string }): Promise<TranscribeResult>;
detectFaces(
request: MediaPathRequest & { sampleFps: number; timesMs?: number[] },
): Promise<DetectFacesResult>;
trackFaces(request: MediaPathRequest & { sampleFps: number }): Promise<TrackFacesResult>;
faceLandmarks(request: MediaPathRequest & { timesMs: number[] }): Promise<FaceLandmarksResult>;
embedFrames(request: MediaPathRequest & { timesMs: number[] }): Promise<EmbedFramesResult>;
classifyContent(
request: MediaPathRequest & { sampleFps: number; extraTimesMs?: number[] },
): Promise<ClassifyContentResult>;
segmentPrompted(
request: MediaPathRequest & { prompt: MediaMarkPrompt; propagate: boolean; outDir: string },
): Promise<SegmentPromptedResult>;
inpaintVideo(
request: MediaPathRequest & { maskDir: string; outPath: string },
): Promise<InpaintVideoResult>;
/**
* Apply face and region redactions frame-accurately and write a near-lossless
* intermediate. Fails closed: a frame it cannot process is an error, not a pass.
*/
redact(
request: MediaPathRequest & { redactions: RedactionInstruction[]; outPath: string },
): Promise<{ outPath: string; framesProcessed: number }>;
/**
* Replace a tracked face with a generated, non-existent identity derived from
* `seed`. Deliberately takes no target-face image.
*/
synthesizeIdentity(
request: MediaPathRequest & { track: BoxKeyframe[]; seed: number; outPath: string },
): Promise<{ outPath: string }>;
audioFeatures(request: MediaPathRequest): Promise<AudioFeaturesResult>;
saliency(request: MediaPathRequest & { timesMs: number[] }): Promise<SaliencyResult>;
}
// ---------------------------------------------------------------------------
// Rendering
// ---------------------------------------------------------------------------
export interface MediaRenderProgress {
stepId: string;
/** Overall progress 0..1. */
progress: number;
}
export interface MediaRenderer {
readonly id: string;
/** Hardware encoders usable on this host, e.g. h264_vaapi, h264_videotoolbox. */
hardwareEncoders(): Promise<string[]>;
/**
* Run every ffmpeg step of the plan. Motion, inpaint, and synthetic identity
* steps must already be satisfied (their outPath exists) or the renderer throws.
*/
render(
plan: RenderPlan,
options: { signal?: AbortSignal; onProgress?: (progress: MediaRenderProgress) => void },
): Promise<void>;
probe(
path: string,
options?: { measureLoudness?: boolean; signal?: AbortSignal },
): Promise<FfprobeReport>;
/** Extract a still frame as PNG (EXIF-free). */
extractFrame(input: {
path: string;
tMs: number;
outPath: string;
width?: number;
signal?: AbortSignal;
}): Promise<void>;
}
export interface MotionRenderRequest {
template: MotionTemplateRef;
brandKit: BrandKit;
/** Resolved font and logo files from the brand kit artifacts. */
assets: { fonts: Array<{ family: string; path: string }>; logoPaths: string[] };
width: number;
height: number;
fps: number;
outPath: string;
signal?: AbortSignal;
}
export interface MotionGraphicsRenderer {
readonly id: string;
templates(): MotionTemplateInfo[];
/** Render an overlay clip with alpha (MOV, qtrle or prores 4444). */
render(request: MotionRenderRequest): Promise<{ path: string; hasAlpha: boolean }>;
}
// ---------------------------------------------------------------------------
// Publishing
// ---------------------------------------------------------------------------
export interface MediaPublishRequest {
bundleDir: string;
profile: PlatformProfile;
publicationId: string;
title: string | null;
caption: string | null;
tags: string[];
signal?: AbortSignal;
}
export interface MediaPublishResult {
status: "published" | "bundled";
externalUrl: string | null;
}
export interface MediaPublisher {
readonly id: string;
publish(request: MediaPublishRequest): Promise<MediaPublishResult>;
}
export interface LinkShortener {
readonly id: string;
shorten(url: string, signal?: AbortSignal): Promise<string>;
}

View File

@ -36,4 +36,8 @@ export const slots = {
web: "web",
browser: "browser",
cloudAgent: "cloud-agent",
mediaInference: "media-inference",
mediaRenderer: "media-renderer",
motionGraphics: "motion-graphics",
mediaPublisher: "media-publisher",
} as const;

View File

@ -1,4 +1,9 @@
import type { ConnectionCatalogItem, ModelTier, SandboxKind } from "@rakazo/contracts";
import type {
ConnectionCatalogItem,
MediaAnalysisKind,
ModelTier,
SandboxKind,
} from "@rakazo/contracts";
export interface AdapterContext {
operationId: string;
@ -524,9 +529,22 @@ export interface BackgroundJobPayloads {
"cloud_agent.poll": { agentId: string };
/** Learn durable user facts from conversations since the profile's last consolidation. */
"profile.consolidate": { userId: string };
/** Enumerate a server-side media source incrementally (metadata only). */
"media.sync": { sourceId: string };
/** Hash the original and build proxy, thumbnails, and keyframes. */
"media.proxy": { assetId: string };
/** Run sidecar analyses; omitted kinds means every analysis not yet at the current version. */
"media.analyze": { assetId: string; kinds?: MediaAnalysisKind[] };
/** Rebuild sessions, themes, and person suggestions for a space. */
"media.cluster": { spaceId: string };
/** Render, probe, and gate one MediaRender. */
"media.render": { renderId: string };
/** Build the export bundle and publish a scheduled MediaPublication. */
"media.publish": { publicationId: string };
}
export type BackgroundJobName = keyof BackgroundJobPayloads;
export type MediaJobName = Extract<BackgroundJobName, `media.${string}`>;
export type BackgroundJob = {
[Name in BackgroundJobName]: {
@ -541,6 +559,8 @@ export type BackgroundJobHandlers = {
[Name in BackgroundJobName]: (payload: BackgroundJobPayloads[Name]) => Promise<void>;
};
export type MediaJobHandlers = Pick<BackgroundJobHandlers, MediaJobName>;
export interface SecretRecord {
id: string;
ciphertext: string;

View File

@ -29,6 +29,8 @@
"@rakazo/core": "workspace:*",
"@rakazo/db": "workspace:*",
"@rakazo/logging": "workspace:*",
"@rakazo/media": "workspace:*",
"@resvg/resvg-js": "2.6.2",
"chat": "4.40.0",
"chat-adapter-lark": "0.3.1",
"chat-adapter-sendblue": "0.2.0",

View File

@ -3,6 +3,7 @@ import type {
AgentRuntime,
BackgroundJobHandlers,
JobPublisher,
MediaJobHandlers,
MessagingSurface,
SandboxProvider,
} from "@rakazo/adapter-kit";
@ -36,6 +37,8 @@ export function createBackgroundJobHandlers(deps: {
deploymentModelKey?: string;
messaging?: MessagingSurface;
cloudAgent?: CloudAgentConnection | null;
/** Media Studio job handlers; absent when media is not configured for this worker. */
media?: MediaJobHandlers;
}): BackgroundJobHandlers {
const deliverMessaging = async (runId?: string) => {
if (!deps.messaging) return;
@ -53,6 +56,7 @@ export function createBackgroundJobHandlers(deps: {
};
return {
...(deps.media ?? mediaDisabledHandlers()),
"run.continue": async (payload) => {
await deps.executor.continueRun(payload.runId, deps.workerId);
// Automatic messaging mirror: once the run's bot messages are durable,
@ -126,3 +130,17 @@ export function createBackgroundJobHandlers(deps: {
},
};
}
function mediaDisabledHandlers(): MediaJobHandlers {
const skip = async () => {
getLogger().warn("media job skipped: Media Studio is not configured on this worker");
};
return {
"media.sync": skip,
"media.proxy": skip,
"media.analyze": skip,
"media.cluster": skip,
"media.render": skip,
"media.publish": skip,
};
}

View File

@ -76,6 +76,24 @@ export * from "./mcp-emulator.js";
export * from "./mcp-oauth.js";
export * from "./mcp-server-tool.js";
export * from "./mcp-transport.js";
export * from "./media-context.js";
export * from "./media-emulator.js";
export * from "./media-emulator-clips.js";
export * from "./media-emulator-source.js";
export * from "./media-factory.js";
export * from "./media-folder-source.js";
export * from "./media-folder-source-probe.js";
export type {
MotionFrameContext,
MotionTemplate,
ResvgMotionGraphicsRendererOptions,
} from "./media-motion-renderer.js";
export {
MOTION_TEMPLATES,
MotionTemplateError,
ResvgMotionGraphicsRenderer,
} from "./media-motion-renderer.js";
export * from "./media-sidecar.js";
export * from "./memory-provider-factory.js";
export * from "./messaging-context.js";
export * from "./messaging-delivery.js";

View File

@ -0,0 +1,281 @@
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { MediaInferenceProvider } from "@rakazo/adapter-kit";
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import type { MediaInferenceOperation } from "@rakazo/contracts";
import { MediaCapabilitiesSchema, MediaInferenceOperationSchema } from "@rakazo/contracts";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { EmulatorMediaInferenceProvider } from "./media-emulator.js";
import { extractRgbFrame, ffmpegAvailable, renderTestClip } from "./media-emulator-clips.js";
import { MEDIA_SIDECAR_RESPONSE_SCHEMAS, SidecarMediaInferenceProvider } from "./media-sidecar.js";
/*
* One contract for every MediaInferenceProvider. The emulator always runs; the
* sidecar runs only in the Docker integration job, which sets
* MEDIA_CONFORMANCE_URL, MEDIA_CONFORMANCE_TOKEN and MEDIA_CONFORMANCE_ROOT (a
* directory inside the sidecar's MEDIA_ROOT, mounted at the same path here).
*/
const WIDTH = 320;
const HEIGHT = 180;
const hasFfmpeg = ffmpegAvailable();
interface Target {
name: string;
/** Directory inside the provider's root where the suite writes fixtures. */
root: () => Promise<string>;
provider: (root: string) => MediaInferenceProvider;
}
const targets: Target[] = [
{
name: "emulator",
root: () => mkdtemp(path.join(tmpdir(), "media-conformance-")),
provider: (root) => new EmulatorMediaInferenceProvider({ root }),
},
];
const sidecarUrl = process.env.MEDIA_CONFORMANCE_URL;
const sidecarToken = process.env.MEDIA_CONFORMANCE_TOKEN;
const sidecarRoot = process.env.MEDIA_CONFORMANCE_ROOT;
if (sidecarUrl && sidecarToken && sidecarRoot) {
targets.push({
name: "sidecar",
root: async () => {
await mkdir(sidecarRoot, { recursive: true });
return mkdtemp(path.join(sidecarRoot, "conformance-"));
},
provider: () => new SidecarMediaInferenceProvider({ baseUrl: sidecarUrl, token: sidecarToken }),
});
}
function requestFor(operation: MediaInferenceOperation, clip: string, root: string) {
const box = { x: 0.25, y: 0.25, w: 0.25, h: 0.25 };
switch (operation) {
case "detectScenes":
case "audioFeatures":
return { path: clip };
case "transcribe":
return { path: clip, language: "en" };
case "detectFaces":
case "trackFaces":
return { path: clip, sampleFps: 2 };
case "faceLandmarks":
case "embedFrames":
case "saliency":
return { path: clip, timesMs: [0, 500, 1000] };
case "classifyContent":
return { path: clip, sampleFps: 2, extraTimesMs: [750] };
case "segmentPrompted":
return {
path: clip,
prompt: { tMs: 500, box },
propagate: true,
outDir: path.join(root, "masks"),
};
case "inpaintVideo":
return {
path: clip,
maskDir: path.join(root, "masks"),
outPath: path.join(root, "inpaint.mp4"),
};
case "redact":
return {
path: clip,
redactions: [
{
keyframes: [
{ tMs: 0, box },
{ tMs: 2000, box },
],
mode: "blur" as const,
paddingPct: 0,
featherPct: 0,
},
],
outPath: path.join(root, "redacted-any.mp4"),
};
case "synthesizeIdentity":
return {
path: clip,
track: [{ tMs: 0, box }],
seed: 7,
outPath: path.join(root, "synthetic.mp4"),
};
}
}
/** Mean absolute RGB difference inside and outside a pixel box (with a guard margin). */
function regionDiff(
a: Buffer,
b: Buffer,
box: { x: number; y: number; w: number; h: number },
margin = 4,
) {
let inside = 0;
let insideCount = 0;
let outside = 0;
let outsideCount = 0;
for (let y = 0; y < HEIGHT; y++) {
for (let x = 0; x < WIDTH; x++) {
const within = x >= box.x && x < box.x + box.w && y >= box.y && y < box.y + box.h;
const near =
x >= box.x - margin &&
x < box.x + box.w + margin &&
y >= box.y - margin &&
y < box.y + box.h + margin;
let diff = 0;
for (let c = 0; c < 3; c++) {
const i = (y * WIDTH + x) * 3 + c;
diff += Math.abs((a[i] ?? 0) - (b[i] ?? 0));
}
if (within) {
inside += diff / 3;
insideCount++;
} else if (!near) {
outside += diff / 3;
outsideCount++;
}
}
}
return { inside: inside / insideCount, outside: outside / outsideCount };
}
for (const target of targets) {
describe.skipIf(!hasFfmpeg)(`MediaInferenceProvider conformance: ${target.name}`, () => {
let root: string;
let clip: string;
let provider: MediaInferenceProvider;
beforeAll(async () => {
root = await target.root();
clip = await renderTestClip(path.join(root, "clip.mp4"), {
seconds: 2,
width: WIDTH,
height: HEIGHT,
});
provider = target.provider(root);
});
afterAll(async () => {
if (root) await rm(root, { recursive: true, force: true });
});
it("reports a capabilities document for every operation", async () => {
const capabilities = MediaCapabilitiesSchema.parse(await provider.capabilities());
expect(Object.keys(capabilities.operations).sort()).toEqual(
[...MediaInferenceOperationSchema.options].sort(),
);
});
it("returns schema-valid output or Unsupported for every operation", async () => {
const capabilities = await provider.capabilities();
// segmentPrompted runs first so inpaintVideo finds its mask directory.
for (const operation of MediaInferenceOperationSchema.options) {
const request = requestFor(operation, clip, root);
const call = provider[operation] as (input: unknown) => Promise<unknown>;
try {
const result = await call.call(provider, request);
const parsed = MEDIA_SIDECAR_RESPONSE_SCHEMAS[operation].safeParse(result);
expect(parsed.success, `${operation}: ${parsed.error?.message}`).toBe(true);
expect(capabilities.operations[operation]?.available, operation).toBe(true);
} catch (error) {
if (!(error instanceof UnsupportedMediaOperationError)) throw error;
expect(error.operation).toBe(operation);
expect(capabilities.operations[operation]?.available, operation).toBe(false);
}
}
});
it("rejects paths outside the media root", async () => {
const outside = await mkdtemp(path.join(tmpdir(), "media-outside-"));
try {
const stray = path.join(outside, "stray.mp4");
await writeFile(stray, "not inside the root");
await expect(provider.detectScenes({ path: stray })).rejects.toThrow();
await expect(
provider.detectScenes({
path: path.join(root, "..", path.basename(outside), "stray.mp4"),
}),
).rejects.toThrow();
await expect(
provider.redact({
path: clip,
redactions: [],
outPath: path.join(outside, "escaped.mp4"),
}),
).rejects.toThrow();
} finally {
await rm(outside, { recursive: true, force: true });
}
});
it("redacts only inside the box", async () => {
const box = { x: 0.25, y: 0.25, w: 0.25, h: 0.25 };
const pixels = { x: 80, y: 44, w: 80, h: 46 };
for (const mode of ["solid", "blur"] as const) {
const outPath = path.join(root, `redacted-${mode}.mp4`);
const result = await provider.redact({
path: clip,
outPath,
redactions: [
{
keyframes: [
{ tMs: 0, box },
{ tMs: 2000, box },
],
mode,
paddingPct: 0,
featherPct: 0,
color: "#102030",
},
],
});
expect(result.framesProcessed).toBeGreaterThan(0);
const source = await extractRgbFrame(clip, 1000);
const redacted = await extractRgbFrame(result.outPath, 1000);
expect(redacted.length).toBe(WIDTH * HEIGHT * 3);
const diff = regionDiff(source, redacted, pixels);
expect(diff.inside, mode).toBeGreaterThan(8);
expect(diff.outside, mode).toBeLessThan(1.5);
}
});
it("honours AbortSignal", async () => {
const aborted = new AbortController();
aborted.abort();
await expect(
provider.detectScenes({ path: clip, signal: aborted.signal }),
).rejects.toMatchObject({ name: "AbortError" });
await expect(provider.capabilities(aborted.signal)).rejects.toMatchObject({
name: "AbortError",
});
const long = await renderTestClip(path.join(root, "long.mp4"), {
seconds: 20,
width: 1280,
height: 720,
});
const controller = new AbortController();
const pending = provider.redact({
path: long,
outPath: path.join(root, "aborted.mp4"),
redactions: [
{
keyframes: [
{ tMs: 0, box: { x: 0.1, y: 0.1, w: 0.3, h: 0.3 } },
{ tMs: 20_000, box: { x: 0.5, y: 0.5, w: 0.3, h: 0.3 } },
],
mode: "blur",
paddingPct: 10,
featherPct: 0,
},
],
signal: controller.signal,
});
setTimeout(() => controller.abort(), 150);
await expect(pending).rejects.toMatchObject({ name: "AbortError" });
});
});
}

View File

@ -0,0 +1,82 @@
import type { AgentRuntime, AgentRuntimeEvent } from "@rakazo/adapter-kit";
import type { PrismaClient } from "@rakazo/db";
import { describe, expect, it, vi } from "vitest";
import { z } from "zod";
import { createMediaModelClient, extractJson } from "./media-context.js";
function runtime(reply: string, compaction = true): AgentRuntime {
return {
describe: () => ({ capabilities: { compaction } }),
run: vi.fn(async function* (): AsyncGenerator<AgentRuntimeEvent> {
yield { type: "done", text: reply };
}),
} as unknown as AgentRuntime;
}
function prisma(consented: boolean): PrismaClient {
return {
aiDataConsent: { findUnique: vi.fn(async () => (consented ? { userId: "u" } : null)) },
} as unknown as PrismaClient;
}
const model = async () => ({ provider: "openai", id: "gpt-test" });
const input = {
spaceId: "s",
userId: "u",
instructions: "Reply with JSON.",
prompt: "Label",
schema: z.object({ title: z.string() }),
};
describe("extractJson", () => {
it("reads fenced, wrapped, and bare JSON", () => {
expect(extractJson('```json\n{"a":1}\n```')).toEqual({ a: 1 });
expect(extractJson('Thinking... {"a":[1,2]} done')).toEqual({ a: [1, 2] });
expect(extractJson("[1]")).toEqual([1]);
expect(extractJson("no json")).toBeUndefined();
expect(extractJson("{broken")).toBeUndefined();
});
});
describe("createMediaModelClient", () => {
it("returns parsed output only with consent for the resolved recipient", async () => {
const withConsent = createMediaModelClient({
prisma: prisma(true),
runtime: runtime('{"title":"Beach"}'),
resolveModel: model,
});
expect(await withConsent.generate(input)).toEqual({ title: "Beach" });
const run = runtime('{"title":"Beach"}');
const withoutConsent = createMediaModelClient({
prisma: prisma(false),
runtime: run,
resolveModel: model,
});
expect(await withoutConsent.generate(input)).toBeNull();
expect(run.run).not.toHaveBeenCalled();
});
it("falls back to null for scripted runtimes, missing models, and invalid replies", async () => {
const scripted = createMediaModelClient({
prisma: prisma(true),
runtime: runtime('{"title":"x"}', false),
resolveModel: model,
});
expect(await scripted.generate(input)).toBeNull();
const noModel = createMediaModelClient({
prisma: prisma(true),
runtime: runtime('{"title":"x"}'),
resolveModel: async () => {
throw new Error("no model");
},
});
expect(await noModel.generate(input)).toBeNull();
const invalid = createMediaModelClient({
prisma: prisma(true),
runtime: runtime('{"name":"x"}'),
resolveModel: model,
});
expect(await invalid.generate(input)).toBeNull();
});
});

View File

@ -0,0 +1,355 @@
import type {
AgentInputImage,
AgentRunModel,
AgentRuntime,
ArtifactStore,
JobPublisher,
LinkShortener,
MediaInferenceProvider,
MediaPublisher,
MediaRenderer,
MediaSourceProvider,
MotionGraphicsRenderer,
} from "@rakazo/adapter-kit";
import type {
AiAlteration,
ClientMediaItem,
CreateMarkInput,
Edl,
EdlError,
FfprobeReport,
GateReport,
IntroVariant,
MarketingCopy,
MarkTreatment,
MediaEditPlan,
MediaMark,
MediaPublication,
MediaRender,
MediaRenderOutput,
MediaRenderQuality,
ModelTier,
PlatformProfile,
Utm,
} from "@rakazo/contracts";
import type { MediaRepos, PrismaClient } from "@rakazo/db";
import { getLogger } from "@rakazo/logging";
import type { CompileSource, ProfileSet } from "@rakazo/media";
import type { z } from "zod";
import { aiRecipient } from "./ai-consent.js";
/*
* Shared wiring for Media Studio services. Composition roots (API, worker)
* build one MediaServiceDeps; every media service, job handler, and agent tool
* receives it instead of reaching for globals.
*/
/** Realtime notifications the web and mobile clients subscribe to per space. */
export type MediaRealtimeEvent =
| { type: "media.source"; sourceId: string; status: string; error?: string | null }
| { type: "media.asset"; assetId: string; ingestState: string }
| { type: "media.collections"; kind: "session" | "theme" }
| { type: "media.people" }
| { type: "media.render"; renderId: string; projectId: string; status: string; progress: number }
| { type: "media.publication"; publicationId: string; status: string };
/**
* Structured model calls on behalf of a user in a space. Returns null instead of
* calling a model when no model is configured, the runtime is scripted, or the
* user has not consented to sharing data with the resolved model recipient.
* Callers must then fall back to deterministic behavior.
*/
export interface MediaModelClient {
generate<T>(input: {
spaceId: string;
userId: string;
/** Short system instructions; the reply must be one JSON document. */
instructions: string;
prompt: string;
images?: AgentInputImage[];
schema: z.ZodType<T>;
tier?: ModelTier;
signal?: AbortSignal;
}): Promise<T | null>;
}
export interface MediaServiceDeps {
prisma: PrismaClient;
repos: MediaRepos;
jobs: JobPublisher;
/** Null when no sidecar is configured; media then stays hidden in the UI. */
inference: MediaInferenceProvider | null;
renderer: MediaRenderer;
motion: MotionGraphicsRenderer;
publisher: MediaPublisher;
shortener: LinkShortener | null;
sources: Partial<Record<"folder" | "emulator", MediaSourceProvider>>;
/** Absolute DATA_DIR/media. Every stored media path is relative to it. */
mediaRoot: string;
/** Builtin plus private platform profiles, cached by the composition root. */
profiles: () => Promise<ProfileSet>;
artifacts: ArtifactStore;
model: MediaModelClient;
publish: (spaceId: string, event: MediaRealtimeEvent) => Promise<void>;
now: () => Date;
}
/** Realtime topic for a space's media events. */
export function mediaTopic(spaceId: string): string {
return `media:${spaceId}`;
}
const MODEL_TIMEOUT_MS = 120_000;
const RUNTIME_FAILURE = /^(?:I hit a problem:|Unknown model )/i;
/** Extract the first JSON object or array from a model reply (models may wrap or think aloud). */
export function extractJson(text: string): unknown {
const trimmed = text.trim();
const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(trimmed);
const body = fenced?.[1] ?? trimmed;
const start = body.search(/[[{]/);
if (start < 0) return undefined;
const open = body[start];
const close = open === "{" ? "}" : "]";
const end = body.lastIndexOf(close);
if (end <= start) return undefined;
try {
return JSON.parse(body.slice(start, end + 1));
} catch {
return undefined;
}
}
export function createMediaModelClient(deps: {
prisma: PrismaClient;
runtime: AgentRuntime;
resolveModel: (scope: {
userId: string;
spaceId: string;
tier?: ModelTier;
}) => Promise<AgentRunModel>;
}): MediaModelClient {
return {
async generate(input) {
if (!deps.runtime.describe().capabilities.compaction) return null;
let model: AgentRunModel;
try {
model = await deps.resolveModel({
userId: input.userId,
spaceId: input.spaceId,
tier: input.tier ?? "fast",
});
} catch {
return null;
}
if (model.provider === "scripted") return null;
const recipient = aiRecipient({
provider: model.provider,
use: "model",
modelId: model.id,
baseUrl: model.baseUrl,
});
if (!recipient) return null;
const consent = await deps.prisma.aiDataConsent.findUnique({
where: {
userId_spaceId_recipientKey: {
userId: input.userId,
spaceId: input.spaceId,
recipientKey: recipient.key,
},
},
});
if (!consent) return null;
const images =
input.images && model.acceptsImages !== false
? input.images.slice(0, model.maxImagesPerPrompt ?? input.images.length)
: undefined;
const runId = `media-${crypto.randomUUID()}`;
const signal = input.signal
? AbortSignal.any([input.signal, AbortSignal.timeout(MODEL_TIMEOUT_MS)])
: AbortSignal.timeout(MODEL_TIMEOUT_MS);
let reply = "";
try {
for await (const event of deps.runtime.run(
{
botId: "media-studio",
threadId: runId,
runId,
prompt: input.prompt,
instructions: input.instructions,
history: [],
tools: [],
model,
...(images && images.length > 0 ? { currentTurnImages: images } : {}),
},
{
operationId: runId,
traceId: runId,
spaceId: input.spaceId,
userId: input.userId,
signal,
},
)) {
if (event.type === "done" && event.text) reply = event.text.trim();
}
} catch (error) {
getLogger().warn("media model call failed", {
reason: error instanceof Error ? error.message : String(error),
});
return null;
}
if (!reply || RUNTIME_FAILURE.test(reply)) return null;
const parsed = input.schema.safeParse(extractJson(reply));
return parsed.success ? parsed.data : null;
},
};
}
// ---------------------------------------------------------------------------
// Service contracts between media slices. Each slice implements its part; the
// API, worker, and agent tools only depend on these interfaces.
// ---------------------------------------------------------------------------
export type CreateMarkInputType = z.infer<typeof CreateMarkInput>;
export interface MediaActor {
spaceId: string;
userId: string;
}
export interface MediaIngestService {
/** Client-pushed metadata (Apple Photos, iOS): upsert, apply deletions, enqueue work. */
pushItems(
actor: MediaActor,
input: {
sourceId: string;
items: ClientMediaItem[];
deletedExternalIds: string[];
syncToken: string | null;
complete: boolean;
},
): Promise<{ created: number; updated: number; unchanged: number; deleted: number }>;
/** Enqueue an incremental sync of a server-side source. */
requestSync(actor: MediaActor, sourceId: string): Promise<void>;
/** An upload finished: move the file into place and enqueue proxy and analysis. */
uploadCompleted(
actor: MediaActor,
input: { assetId: string; tempPath: string; sha256: string; fileName: string },
): Promise<void>;
recluster(actor: MediaActor): Promise<void>;
/** Remove a source with its assets and every derived file. */
removeSource(actor: MediaActor, sourceId: string): Promise<void>;
}
export interface MediaPreparedSources {
sources: Map<string, CompileSource>;
/** Alterations applied while preparing (inpaint, synthetic identity, voice). */
aiAlterations: AiAlteration[];
/** Redactions that could not be honored and fell back, for the plan rationale. */
fallbacks: string[];
}
export interface MediaPrivacyService {
/**
* Resolve every clip's source file and apply all face, mark, and region
* redactions from the EDL plus person and mark defaults. Fails closed: a
* redaction that cannot be applied throws instead of returning an unredacted source.
*/
prepareSources(input: {
actor: MediaActor;
projectId: string;
edl: Edl;
quality: MediaRenderQuality;
workDir: string;
signal?: AbortSignal;
}): Promise<MediaPreparedSources>;
/** Re-detect on the rendered output and evaluate the full gate. */
gate(input: {
actor: MediaActor;
projectId: string;
edl: Edl;
profile: PlatformProfile;
outputPath: string;
ffprobe: FfprobeReport;
copy: { title: string | null; caption: string | null; tags: string[] };
bundleHasDisclosure: boolean;
autoFixRound: number;
signal?: AbortSignal;
}): Promise<GateReport>;
/** Create a mark: segment from the prompt, propagate, score, and suggest a fallback. */
createMark(actor: MediaActor, input: CreateMarkInputType): Promise<MediaMark>;
updateMark(
actor: MediaActor,
input: {
markId: string;
treatment?: MarkTreatment;
label?: string | null;
personId?: string | null;
},
): Promise<MediaMark>;
/** Delete a person, their biometric data, and their files. */
removePerson(actor: MediaActor, personId: string): Promise<void>;
}
export interface MediaRenderService {
start(
actor: MediaActor,
input: { planId: string; quality: MediaRenderQuality },
): Promise<MediaRender>;
cancel(actor: MediaActor, renderId: string): Promise<MediaRender>;
/** Absolute path of a finished output for streaming, or null. */
outputPath(actor: MediaActor, renderId: string, fileName: string): Promise<string | null>;
}
export interface MediaMarketingService {
autoPlan(
actor: MediaActor,
input: { projectId: string; profileIds?: string[] },
): Promise<MediaEditPlan[]>;
submitPlan(
actor: MediaActor,
input: {
projectId: string;
edl: unknown;
rationale: string | null;
createdBy: "agent" | "user";
},
): Promise<{ ok: true; plan: MediaEditPlan } | { ok: false; errors: EdlError[] }>;
introVariants(
actor: MediaActor,
input: { brandKitId: string; count: number },
): Promise<IntroVariant[]>;
thumbnails(
actor: MediaActor,
input: { renderId: string; count: number },
): Promise<MediaRenderOutput[]>;
copy(actor: MediaActor, input: { renderId: string; language: string }): Promise<MarketingCopy>;
createPublication(
actor: MediaActor,
input: {
renderId: string;
destination: string;
targetUrl: string | null;
utm: Utm | null;
title: string | null;
caption: string | null;
tags: string[];
thumbnailVariant: string | null;
scheduledAt: string | null;
},
): Promise<MediaPublication>;
bundle(actor: MediaActor, publicationId: string): Promise<MediaPublication>;
cancelPublication(actor: MediaActor, publicationId: string): Promise<MediaPublication>;
importMetricsCsv(
actor: MediaActor,
input: { mappingId: string; csv: string },
): Promise<{ imported: number; skipped: Array<{ row: number; reason: string }> }>;
}
export interface MediaServices {
deps: MediaServiceDeps;
ingest: MediaIngestService;
privacy: MediaPrivacyService;
render: MediaRenderService;
marketing: MediaMarketingService;
}

View File

@ -0,0 +1,66 @@
import { spawnSync } from "node:child_process";
import { runMediaTool } from "./media-folder-source-probe.js";
/*
* Synthetic test clips (ffmpeg testsrc2 + sine) for media tests across the
* codebase. Name the output with media-emulator markers to steer emulator
* results, e.g. `clip__faces-1__.mp4`.
*/
export function ffmpegAvailable(ffmpegPath = "ffmpeg"): boolean {
return spawnSync(ffmpegPath, ["-version"], { stdio: "ignore" }).status === 0;
}
export interface TestClipOptions {
seconds?: number;
width?: number;
height?: number;
fps?: number;
audio?: boolean;
ffmpegPath?: string;
signal?: AbortSignal;
}
export async function renderTestClip(outPath: string, options: TestClipOptions = {}) {
const { seconds = 2, width = 320, height = 180, fps = 25, audio = true } = options;
const args = [
"-v",
"error",
"-y",
"-f",
"lavfi",
"-i",
`testsrc2=s=${width}x${height}:r=${fps}:d=${seconds}`,
];
if (audio)
args.push("-f", "lavfi", "-i", `sine=f=440:sample_rate=48000:d=${seconds}`, "-shortest");
args.push("-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p");
if (audio) args.push("-c:a", "aac");
args.push(outPath);
await runMediaTool(options.ffmpegPath ?? "ffmpeg", args, { signal: options.signal });
return outPath;
}
/** Decode one frame at `tMs` as packed RGB24 bytes. */
export async function extractRgbFrame(
file: string,
tMs: number,
options: { ffmpegPath?: string } = {},
): Promise<Buffer> {
const { stdout } = await runMediaTool(options.ffmpegPath ?? "ffmpeg", [
"-v",
"error",
"-i",
file,
"-ss",
(tMs / 1000).toFixed(3),
"-frames:v",
"1",
"-f",
"rawvideo",
"-pix_fmt",
"rgb24",
"-",
]);
return stdout;
}

View File

@ -0,0 +1,208 @@
import { randomUUID } from "node:crypto";
import { mkdir, rename, rm, stat } from "node:fs/promises";
import path from "node:path";
import type {
MediaSourceItem,
MediaSourceListRequest,
MediaSourceListResult,
MediaSourceProvider,
} from "@rakazo/adapter-kit";
import { PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import { z } from "zod";
import type { MediaToolPaths } from "./media-folder-source-probe.js";
import { runMediaTool } from "./media-folder-source-probe.js";
/*
* Deterministic synthetic library for tests and the scripted runtime. Config
* `{ count, seed }` describes `count` assets split into three capture sessions
* on separate days in three cities (more than 2 km apart), alternating two
* visual themes (`__theme-a__` testsrc2, `__theme-b__` SMPTE bars). Every fifth
* asset is a photo. File names carry media-emulator markers:
* index % 10 == 0 __faces-1____person-a__
* index % 10 == 1 __faces-1____person-b__
* index == 2 __nsfw-explicit_nudity-0.9__
* index % 4 == 3 __speech__ (videos only)
* index % 10 == 6 __cut-2000__ (videos only)
* `resolve` renders the clip on first use (4 s, 320x180, ffmpeg testsrc2 or
* smptehdbars plus a sine tone) and caches it under `fixturesDir/<seed>/`.
*/
const ConfigSchema = z.object({
count: z.number().int().min(1).max(100_000),
seed: z.number().int().min(0).max(2_147_483_647),
});
const CLIP_MS = 4000;
const CLIP_SIZE = { width: 320, height: 180, fps: 25 };
const SESSIONS = [
{ lat: 52.52, lon: 13.405 },
{ lat: 53.5511, lon: 9.9937 },
{ lat: 48.1374, lon: 11.5755 },
] as const;
export interface EmulatorAsset {
index: number;
fileName: string;
kind: "video" | "photo";
theme: "a" | "b";
session: number;
capturedAt: Date;
gpsLat: number;
gpsLon: number;
}
/** The deterministic description of asset `index` in an emulator library. */
export function emulatorAsset(seed: number, count: number, index: number): EmulatorAsset {
const kind = index % 5 === 4 ? "photo" : "video";
const theme = index % 2 === 0 ? "a" : "b";
const session = Math.min(SESSIONS.length - 1, Math.floor((index * SESSIONS.length) / count));
const firstInSession = Math.ceil((session * count) / SESSIONS.length);
const day = Date.UTC(2026, 4, 2 + (seed % 28) + session * 2, 8 + session);
const capturedAt = new Date(day + (index - firstInSession) * 3 * 60_000);
const place = SESSIONS[session] ?? SESSIONS[0];
const jitter = ((index * 7919 + seed) % 100) / 100_000;
const markers = [`__theme-${theme}__`];
if (index % 10 === 0) markers.push("__faces-1____person-a__");
if (index % 10 === 1) markers.push("__faces-1____person-b__");
if (index === 2) markers.push("__nsfw-explicit_nudity-0.9__");
if (kind === "video" && index % 4 === 3) markers.push("__speech__");
if (kind === "video" && index % 10 === 6) markers.push("__cut-2000__");
const ext = kind === "photo" ? "jpg" : "mp4";
return {
index,
fileName: `emu-${seed}-${String(index).padStart(5, "0")}${markers.join("")}.${ext}`,
kind,
theme,
session,
capturedAt,
gpsLat: Math.round((place.lat + jitter) * 1e5) / 1e5,
gpsLon: Math.round((place.lon + jitter) * 1e5) / 1e5,
};
}
function iso6709(lat: number, lon: number): string {
const sign = (value: number, width: number) =>
`${value < 0 ? "-" : "+"}${Math.abs(value).toFixed(4).padStart(width, "0")}`;
return `${sign(lat, 7)}${sign(lon, 8)}/`;
}
export interface EmulatorMediaSourceOptions extends MediaToolPaths {
/** Cache for rendered clips; must lie inside the media root to be analysable. */
fixturesDir: string;
}
export class EmulatorMediaSourceProvider implements MediaSourceProvider {
readonly kind = "emulator" as const;
constructor(private readonly options: EmulatorMediaSourceOptions) {}
async list(request: MediaSourceListRequest): Promise<MediaSourceListResult> {
request.signal?.throwIfAborted();
const { count, seed } = ConfigSchema.parse(request.config);
const start = request.cursor ? decodeIndex(request.cursor) : 0;
const end = Math.min(count, start + Math.max(1, Math.floor(request.limit)));
const items: MediaSourceItem[] = [];
for (let index = start; index < end; index++) {
const asset = emulatorAsset(seed, count, index);
items.push({
externalId: asset.fileName,
kind: asset.kind,
capturedAt: asset.capturedAt,
durationMs: asset.kind === "video" ? CLIP_MS : null,
width: CLIP_SIZE.width,
height: CLIP_SIZE.height,
fps: asset.kind === "video" ? CLIP_SIZE.fps : null,
gpsLat: asset.gpsLat,
gpsLon: asset.gpsLon,
deviceModel: "Emulator",
albumNames: [],
isFavorite: index % 9 === 0,
fileName: asset.fileName,
sizeBytes: null,
revision: `emulator-v1-${seed}`,
});
}
return {
items,
nextCursor: end < count ? Buffer.from(String(end)).toString("base64url") : null,
};
}
async resolve(config: Record<string, unknown>, externalId: string): Promise<string> {
const { count, seed } = ConfigSchema.parse(config);
const match = /^emu-(\d+)-(\d{5})/.exec(externalId);
const index = match ? Number(match[2]) : -1;
if (!match || Number(match[1]) !== seed || index >= count)
throw new PathOutsideRootError(externalId);
const asset = emulatorAsset(seed, count, index);
if (asset.fileName !== externalId) throw new PathOutsideRootError(externalId);
await mkdir(path.join(this.options.fixturesDir, String(seed)), { recursive: true });
const target = await resolveInside(this.options.fixturesDir, `${seed}/${asset.fileName}`);
if (await exists(target)) return target;
const partial = path.join(
path.dirname(target),
`.partial-${randomUUID()}.${asset.kind === "photo" ? "jpg" : "mp4"}`,
);
try {
await runMediaTool(this.options.ffmpegPath ?? "ffmpeg", renderArgs(asset, partial));
await rename(partial, target);
} finally {
await rm(partial, { force: true });
}
return target;
}
}
function renderArgs(asset: EmulatorAsset, out: string): string[] {
const { width, height, fps } = CLIP_SIZE;
const pattern = asset.theme === "a" ? "testsrc2" : "smptehdbars";
const video = `${pattern}=s=${width}x${height}:r=${fps}:d=${CLIP_MS / 1000}`;
const metadata = [
"-metadata",
`creation_time=${asset.capturedAt.toISOString()}`,
"-metadata",
`location=${iso6709(asset.gpsLat, asset.gpsLon)}`,
];
if (asset.kind === "photo")
return ["-v", "error", "-y", "-f", "lavfi", "-i", video, "-frames:v", "1", out];
return [
"-v",
"error",
"-y",
"-f",
"lavfi",
"-i",
video,
"-f",
"lavfi",
"-i",
`sine=f=${220 + (asset.index % 12) * 40}:sample_rate=48000:d=${CLIP_MS / 1000}`,
"-shortest",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
"-c:a",
"aac",
...metadata,
"-movflags",
"+faststart",
out,
];
}
function decodeIndex(cursor: string): number {
const value = Number(Buffer.from(cursor, "base64url").toString("utf8"));
if (!Number.isInteger(value) || value < 0) throw new Error("invalid emulator cursor");
return value;
}
async function exists(file: string): Promise<boolean> {
try {
return (await stat(file)).isFile();
} catch {
return false;
}
}

View File

@ -0,0 +1,335 @@
import { copyFile, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import { cosineSimilarity, haversineKm } from "@rakazo/media";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
buildRedactionGraph,
EmulatorMediaInferenceProvider,
parseEmulatorMarkers,
redactionRegions,
} from "./media-emulator.js";
import { ffmpegAvailable, renderTestClip } from "./media-emulator-clips.js";
import { EmulatorMediaSourceProvider, emulatorAsset } from "./media-emulator-source.js";
import { probeMedia } from "./media-folder-source-probe.js";
const hasFfmpeg = ffmpegAvailable();
let root: string;
let provider: EmulatorMediaInferenceProvider;
beforeAll(async () => {
root = await mkdtemp(path.join(tmpdir(), "media-emulator-"));
provider = new EmulatorMediaInferenceProvider({ root });
});
afterAll(async () => {
await rm(root, { recursive: true, force: true });
});
/** A non-media file: the emulator falls back to a 10 s clip, so no ffmpeg is needed. */
async function fixture(name: string, content = "synthetic bytes") {
const file = path.join(root, name);
await writeFile(file, content);
return file;
}
describe("parseEmulatorMarkers", () => {
it("reads every documented marker", () => {
expect(
parseEmulatorMarkers(
"/x/clip__faces-2____person-a____noface-gap-1000-2500____nsfw-explicit_nudity-0.9____nsfw-violence-0.4____cut-1500____cut-3000____speech____theme-b____inpaintq-0.3__.mp4",
),
).toEqual({
faces: 2,
person: "a",
noFaceGaps: [{ startMs: 1000, endMs: 2500 }],
nsfw: { explicit_nudity: 0.9, violence: 0.4 },
cutsMs: [1500, 3000],
speech: true,
theme: "b",
inpaintQuality: 0.3,
});
expect(parseEmulatorMarkers("__nsfw-unknown-0.5____nsfw-violence-7__.mp4").nsfw).toEqual({});
});
});
describe("redaction regions", () => {
it("holds the union of neighbouring keyframes and bounds the graph", () => {
const regions = redactionRegions(
{
keyframes: [
{ tMs: 1000, box: { x: 0.5, y: 0.5, w: 0.1, h: 0.1 } },
{ tMs: 0, box: { x: 0.1, y: 0.1, w: 0.1, h: 0.1 } },
],
mode: "blur",
paddingPct: 0,
featherPct: 0,
},
1000,
1000,
);
expect(regions).toEqual([
{ startMs: 0, endMs: 1000, box: { x: 100, y: 100, w: 500, h: 500 } },
{ startMs: 1000, endMs: 1100, box: { x: 500, y: 500, w: 100, h: 100 } },
]);
const keyframes = Array.from({ length: 5000 }, (_, i) => ({
tMs: i * 40,
box: { x: (i % 50) / 100, y: 0.2, w: 0.1, h: 0.1 },
}));
const many = redactionRegions(
{ keyframes, mode: "pixelate", paddingPct: 25, featherPct: 0 },
1920,
1080,
);
expect(many.length).toBeLessThanOrEqual(400);
expect(many[0]?.startMs).toBe(0);
expect(many.at(-1)?.endMs).toBe(4999 * 40 + 100);
const graph = buildRedactionGraph(
[{ keyframes, mode: "solid", paddingPct: 0, featherPct: 0, color: "#ff0000" }],
1920,
1080,
);
expect(graph.filter).toContain("color=0xff0000@1:t=fill");
});
});
describe("EmulatorMediaInferenceProvider", () => {
it("is quiet without markers", async () => {
const file = await fixture("plain.mp4");
const scenes = await provider.detectScenes({ path: file });
expect(scenes.scenes).toEqual([{ startMs: 0, endMs: 10_000 }]);
expect(scenes.shots).toHaveLength(1);
expect((await provider.trackFaces({ path: file, sampleFps: 2 })).tracks).toEqual([]);
const faces = await provider.detectFaces({ path: file, sampleFps: 1 });
expect(faces.frames).toHaveLength(10);
expect(faces.frames.every((frame) => frame.faces.length === 0)).toBe(true);
const classified = await provider.classifyContent({ path: file, sampleFps: 2 });
expect(classified.frames).toHaveLength(20);
for (const frame of classified.frames)
expect(Object.values(frame.scores).every((score) => score === 0.01)).toBe(true);
expect(await provider.transcribe({ path: file })).toEqual({ language: null, words: [] });
});
it("forces faces, gaps, cuts, classifier scores and speech from the file name", async () => {
const file = await fixture(
"rich__faces-2____noface-gap-2000-3000____cut-4000____nsfw-explicit_nudity-0.9____speech__.mp4",
);
const tracks = await provider.trackFaces({ path: file, sampleFps: 2 });
expect(tracks.tracks.map((track) => track.trackId)).toEqual(["track-1", "track-2"]);
const times = tracks.tracks[0]?.keyframes.map((keyframe) => keyframe.tMs) ?? [];
expect(times).toContain(1500);
expect(times).not.toContain(2000);
expect(times).not.toContain(2500);
expect(times).toContain(3000);
const gapFrame = await provider.detectFaces({
path: file,
sampleFps: 1,
timesMs: [2500, 3500],
});
expect(gapFrame.frames.map((frame) => frame.faces.length)).toEqual([0, 2]);
expect((await provider.detectScenes({ path: file })).scenes).toEqual([
{ startMs: 0, endMs: 4000 },
{ startMs: 4000, endMs: 10_000 },
]);
const classified = await provider.classifyContent({
path: file,
sampleFps: 0.5,
extraTimesMs: [4000, 4001],
});
expect(classified.frames.map((frame) => frame.tMs)).toEqual([0, 2000, 4000, 4001, 6000, 8000]);
for (const frame of classified.frames) {
expect(frame.scores.explicit_nudity).toBe(0.9);
expect(frame.scores.violence).toBe(0.01);
}
const english = await provider.transcribe({ path: file });
const german = await provider.transcribe({ path: file, language: "de-DE" });
expect(english.language).toBe("en");
expect(english.words.map((word) => word.text).slice(0, 2)).toEqual(["welcome", "back"]);
expect(german.language).toBe("de");
expect(german.words[0]).toEqual({
startMs: 400,
endMs: 720,
text: "willkommen",
confidence: 0.95,
});
});
it("is deterministic per file content and keeps boxes stable", async () => {
const file = await fixture("same__faces-1__.mp4");
const copy = path.join(root, "copy__faces-1__.mp4");
await copyFile(file, copy);
const a = await provider.detectFaces({ path: file, sampleFps: 4 });
const b = await provider.detectFaces({ path: file, sampleFps: 4 });
const c = await provider.detectFaces({ path: copy, sampleFps: 4 });
expect(a).toEqual(b);
expect(a).toEqual(c);
const boxes = a.frames.map((frame) => frame.faces[0]?.box);
expect(Math.abs((boxes.at(-1)?.x ?? 0) - (boxes[0]?.x ?? 0))).toBeLessThan(0.05);
});
it("embeds people and themes consistently", async () => {
const personA1 = await fixture("one__faces-1____person-a__.mp4", "first");
const personA2 = await fixture("two__faces-1____person-a__.mp4", "second");
const personB = await fixture("three__faces-1____person-b__.mp4", "third");
const embed = async (file: string) =>
(await provider.trackFaces({ path: file, sampleFps: 1 })).tracks[0]?.embedding ?? [];
expect(cosineSimilarity(await embed(personA1), await embed(personA2))).toBeCloseTo(1, 5);
expect(cosineSimilarity(await embed(personA1), await embed(personB))).toBeLessThan(0.8);
const themeA1 = await fixture("x__theme-a__.mp4", "x");
const themeA2 = await fixture("y__theme-a__.mp4", "y");
const themeB = await fixture("z__theme-b__.mp4", "z");
const mean = async (file: string) =>
(await provider.embedFrames({ path: file, timesMs: [0, 1000] })).vectors[0] ?? [];
expect(cosineSimilarity(await mean(themeA1), await mean(themeA2))).toBeGreaterThan(0.95);
expect(cosineSimilarity(await mean(themeA1), await mean(themeB))).toBeLessThan(0.85);
});
it("never synthesizes identities", async () => {
const file = await fixture("synth.mp4");
const capabilities = await provider.capabilities();
expect(capabilities.operations.synthesizeIdentity).toMatchObject({
available: false,
license: null,
});
expect(capabilities.operations.inpaintVideo).toMatchObject({
available: true,
experimental: true,
});
await expect(
provider.synthesizeIdentity({
path: file,
track: [],
seed: 1,
outPath: path.join(root, "s.mp4"),
}),
).rejects.toBeInstanceOf(UnsupportedMediaOperationError);
});
it("rejects paths outside its root", async () => {
await expect(provider.detectScenes({ path: "/etc/hostname" })).rejects.toThrow(
"outside the allowed root",
);
await expect(provider.detectScenes({ path: "relative.mp4" })).rejects.toThrow();
});
it.skipIf(!hasFfmpeg)("writes masks and inpaints with the reported quality", async () => {
const clip = await renderTestClip(path.join(root, "mark__inpaintq-0.4__.mp4"), {
seconds: 1,
audio: false,
});
const masks = await provider.segmentPrompted({
path: clip,
prompt: { tMs: 0, point: { x: 0.5, y: 0.5 } },
propagate: true,
outDir: path.join(root, "marks/m1"),
});
expect((await readdir(masks.maskDir)).sort()).toEqual(["0.png", "500.png"]);
expect(masks.keyframes[0]?.box).toEqual({ x: 0.44, y: 0.44, w: 0.12, h: 0.12 });
const inpainted = await provider.inpaintVideo({
path: clip,
maskDir: masks.maskDir,
outPath: path.join(root, "marks/m1.mp4"),
});
expect(inpainted.quality.score).toBe(0.4);
expect((await probeMedia(inpainted.outPath)).durationMs).toBeGreaterThan(900);
});
it.skipIf(!hasFfmpeg)("renders very large redaction graphs through a script file", async () => {
const clip = await renderTestClip(path.join(root, "busy.mp4"), { seconds: 1, audio: false });
const instruction = (offset: number) => ({
keyframes: Array.from({ length: 2000 }, (_, i) => ({
tMs: i,
box: { x: offset + (i % 40) / 400, y: 0.1, w: 0.1, h: 0.1 },
})),
mode: "blur" as const,
paddingPct: 0,
featherPct: 0,
});
const instructions = [instruction(0), instruction(0.3), instruction(0.6)];
expect(buildRedactionGraph(instructions, 320, 180).filter.length).toBeGreaterThan(100_000);
const result = await provider.redact({
path: clip,
redactions: instructions,
outPath: path.join(root, "busy-redacted.mp4"),
});
expect(result.framesProcessed).toBe(25);
});
it.skipIf(!hasFfmpeg)("measures loudness with ebur128", async () => {
const clip = await renderTestClip(path.join(root, "tone.mp4"), { seconds: 2 });
const features = await provider.audioFeatures({ path: clip });
expect(features.integratedLufs).toBeLessThan(-10);
expect(features.integratedLufs).toBeGreaterThan(-40);
expect(features.energy.length).toBeGreaterThan(0);
expect(features.beatsMs.length).toBeGreaterThan(0);
const silent = await renderTestClip(path.join(root, "mute.mp4"), { seconds: 1, audio: false });
expect(await provider.audioFeatures({ path: silent })).toMatchObject({
integratedLufs: null,
silences: [{ startMs: 0, endMs: 1000 }],
});
});
});
describe("EmulatorMediaSourceProvider", () => {
const config = { count: 30, seed: 4 };
it("describes three sessions, two themes and the marker assets", async () => {
const source = new EmulatorMediaSourceProvider({ fixturesDir: path.join(root, "library") });
const first = await source.list({ config, cursor: null, limit: 12 });
const second = await source.list({ config, cursor: first.nextCursor, limit: 100 });
expect(second.nextCursor).toBeNull();
const items = [...first.items, ...second.items];
expect(items).toHaveLength(30);
expect(new Set(items.map((item) => item.externalId)).size).toBe(30);
expect(await source.list({ config, cursor: null, limit: 12 })).toEqual(first);
const assets = items.map((_, index) => emulatorAsset(config.seed, config.count, index));
expect(new Set(assets.map((asset) => asset.session))).toEqual(new Set([0, 1, 2]));
expect(new Set(assets.map((asset) => asset.theme))).toEqual(new Set(["a", "b"]));
const sessionStarts = [0, 1, 2].map(
(session) => assets.find((asset) => asset.session === session)!,
);
expect(
haversineKm(
{ lat: sessionStarts[0]!.gpsLat, lon: sessionStarts[0]!.gpsLon },
{ lat: sessionStarts[1]!.gpsLat, lon: sessionStarts[1]!.gpsLon },
),
).toBeGreaterThan(2);
const names = items.map((item) => item.externalId).join(" ");
expect(names).toContain("__faces-1____person-a__");
expect(names).toContain("__faces-1____person-b__");
expect(names).toContain("__nsfw-explicit_nudity-0.9__");
expect(names).toContain("__speech__");
expect(items.some((item) => item.kind === "photo")).toBe(true);
});
it("rejects unknown external ids", async () => {
const source = new EmulatorMediaSourceProvider({ fixturesDir: path.join(root, "library") });
for (const bad of [
"../etc/passwd",
"emu-4-00099__theme-b__.mp4",
"emu-5-00000__theme-a__.mp4",
"emu-4-00000.mp4",
])
await expect(source.resolve(config, bad), bad).rejects.toThrow();
});
it.skipIf(!hasFfmpeg)("renders and caches clips on demand", async () => {
const source = new EmulatorMediaSourceProvider({ fixturesDir: path.join(root, "library") });
const asset = emulatorAsset(config.seed, config.count, 0);
const file = await source.resolve(config, asset.fileName);
expect(path.basename(file)).toBe(asset.fileName);
const probed = await probeMedia(file);
expect(probed).toMatchObject({ durationMs: 4000, width: 320, height: 180, hasAudio: true });
expect(probed.capturedAt?.toISOString()).toBe(asset.capturedAt.toISOString());
expect(await source.resolve(config, asset.fileName)).toBe(file);
const tracks = await provider.trackFaces({ path: file, sampleFps: 1 });
expect(tracks.tracks).toHaveLength(1);
const photo = emulatorAsset(config.seed, config.count, 4);
expect(photo.kind).toBe("photo");
expect((await probeMedia(await source.resolve(config, photo.fileName))).width).toBe(320);
});
});

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,53 @@
import { homedir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
createMediaInferenceProvider,
createMediaSourceProvider,
mediaConfigFromEnv,
resolveMediaInferenceKind,
} from "./media-factory.js";
const worker = {
MEDIA_WORKER_URL: "http://127.0.0.1:8790",
MEDIA_WORKER_TOKEN: "placeholder-token-placeholder-token-00",
};
describe("media factory", () => {
it("reads one generic worker connection and derives paths", () => {
expect(mediaConfigFromEnv({ DATA_DIR: "/srv/evoo", ...worker })).toEqual({
enabled: true,
workerUrl: worker.MEDIA_WORKER_URL,
workerToken: worker.MEDIA_WORKER_TOKEN,
profilesDir: path.join(homedir(), ".evoo/media-profiles"),
mediaRoot: "/srv/evoo/media",
});
expect(
mediaConfigFromEnv({ DATA_DIR: "/srv/evoo", MEDIA_PROFILES_DIR: "/etc/evoo/profiles" }),
).toMatchObject({ enabled: false, workerUrl: null, profilesDir: "/etc/evoo/profiles" });
});
it("selects the sidecar only with URL and token", () => {
expect(resolveMediaInferenceKind(worker)).toBe("sidecar");
expect(createMediaInferenceProvider(worker)?.id).toBe("sidecar");
expect(resolveMediaInferenceKind({ MEDIA_WORKER_URL: worker.MEDIA_WORKER_URL })).toBeNull();
expect(createMediaInferenceProvider({})).toBeNull();
});
it("pins the emulator under the scripted runtime", () => {
expect(resolveMediaInferenceKind({ ...worker, AGENT_RUNTIME: "scripted" })).toBe("emulator");
expect(createMediaInferenceProvider({ AGENT_RUNTIME: "scripted" })?.id).toBe("emulator");
expect(mediaConfigFromEnv({ AGENT_RUNTIME: "scripted" }).enabled).toBe(true);
// pin-test-env sets AGENT_RUNTIME=scripted for every unit test.
expect(createMediaInferenceProvider()?.id).toBe("emulator");
});
it("creates server-side sources only", () => {
expect(createMediaSourceProvider("folder", {}).kind).toBe("folder");
expect(createMediaSourceProvider("emulator", { AGENT_RUNTIME: "scripted" }).kind).toBe(
"emulator",
);
expect(() => createMediaSourceProvider("emulator", {})).toThrow();
expect(() => createMediaSourceProvider("apple_photos", {})).toThrow();
});
});

View File

@ -0,0 +1,90 @@
import { homedir } from "node:os";
import path from "node:path";
import type { MediaInferenceProvider, MediaSourceProvider } from "@rakazo/adapter-kit";
import { expandHome } from "@rakazo/media/node";
import { EmulatorMediaInferenceProvider } from "./media-emulator.js";
import { EmulatorMediaSourceProvider } from "./media-emulator-source.js";
import { FolderMediaSourceProvider } from "./media-folder-source.js";
import { SidecarMediaInferenceProvider } from "./media-sidecar.js";
export interface MediaConfig {
/** Media Studio runs: a sidecar is configured or the scripted runtime uses the emulator. */
enabled: boolean;
workerUrl: string | null;
workerToken: string | null;
profilesDir: string;
/** `${DATA_DIR}/media`, absolute. Every derived file and sidecar path lives here. */
mediaRoot: string;
}
export type MediaInferenceKind = "sidecar" | "emulator";
/** Emulators replace the sidecar whenever the scripted runtime is pinned (tests, e2e). */
export function mediaEmulatorEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
return env.AGENT_RUNTIME === "scripted";
}
export function mediaConfigFromEnv(env: NodeJS.ProcessEnv = process.env): MediaConfig {
const workerUrl = env.MEDIA_WORKER_URL?.trim() || null;
const workerToken = env.MEDIA_WORKER_TOKEN?.trim() || null;
return {
enabled: mediaEmulatorEnabled(env) || Boolean(workerUrl && workerToken),
workerUrl,
workerToken,
profilesDir: path.resolve(
expandHome(env.MEDIA_PROFILES_DIR?.trim() || "~/.evoo/media-profiles", homedir()),
),
mediaRoot: path.resolve(env.DATA_DIR ?? "./data", "media"),
};
}
export function resolveMediaInferenceKind(
env: NodeJS.ProcessEnv = process.env,
): MediaInferenceKind | null {
if (mediaEmulatorEnabled(env)) return "emulator";
const config = mediaConfigFromEnv(env);
return config.workerUrl && config.workerToken ? "sidecar" : null;
}
/** The configured inference provider, or null when Media Studio is not set up. */
export function createMediaInferenceProvider(
env: NodeJS.ProcessEnv = process.env,
): MediaInferenceProvider | null {
const config = mediaConfigFromEnv(env);
switch (resolveMediaInferenceKind(env)) {
case "emulator":
return new EmulatorMediaInferenceProvider({ root: config.mediaRoot });
case "sidecar":
return new SidecarMediaInferenceProvider({
baseUrl: config.workerUrl as string,
token: config.workerToken as string,
});
default:
return null;
}
}
/** Emulator clips are cached inside the media root so inference can read them. */
export function emulatorFixturesDir(env: NodeJS.ProcessEnv = process.env): string {
return path.join(mediaConfigFromEnv(env).mediaRoot, "emulator-library");
}
/**
* Server-side source adapter for a persisted MediaSource kind. Client-pushed
* kinds (apple_photos, ios_library) have no server adapter.
*/
export function createMediaSourceProvider(
kind: string,
env: NodeJS.ProcessEnv = process.env,
): MediaSourceProvider {
switch (kind) {
case "folder":
return new FolderMediaSourceProvider();
case "emulator":
if (!mediaEmulatorEnabled(env)) break;
return new EmulatorMediaSourceProvider({ fixturesDir: emulatorFixturesDir(env) });
default:
break;
}
throw new Error(`No server-side media source for "${kind}". Use folder.`);
}

View File

@ -0,0 +1,599 @@
import { spawn } from "node:child_process";
import { access, mkdir, rm, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import type { MediaRenderer, MediaRenderProgress } from "@rakazo/adapter-kit";
import type { FfprobeReport, RenderPlan, RenderStep } from "@rakazo/contracts";
import { getLogger } from "@rakazo/logging";
import { assertInside } from "@rakazo/media/node";
import { MediaToolMissingError, runMediaTool } from "./media-folder-source-probe.js";
/*
* Local ffmpeg executor for compiled render plans. Every process is spawned
* with an argv array and no shell. Plans come from compileRenderPlan, which
* already escapes every value; this module only adds runtime options
* (progress, hardware device) and the loudness measurement pass.
*
* Loudness: when the render step carries a single `loudnorm` with
* linear=true and no measured values, a first pass measures the program
* (print_format=json, null muxer) and the filter is rewritten with the
* measured_* options, so the second pass normalizes linearly to the target.
* When the measurement cannot be parsed the step runs single-pass (dynamic).
*/
const HARDWARE_CANDIDATES = [
"h264_vaapi",
"hevc_vaapi",
"h264_videotoolbox",
"hevc_videotoolbox",
"h264_nvenc",
"hevc_nvenc",
] as const;
const DEFAULT_VAAPI_DEVICE = "/dev/dri/renderD128";
const DEFAULT_KILL_GRACE_MS = 2000;
const STDERR_TAIL_BYTES = 64 * 1024;
export interface FfmpegMediaRendererOptions {
ffmpegPath?: string;
ffprobePath?: string;
/** When set, plan work dirs and probed or extracted paths must lie inside it. */
root?: string;
vaapiDevice?: string;
/** Time between SIGTERM and SIGKILL when a render is aborted. */
killGraceMs?: number;
}
function abortError(signal: AbortSignal | undefined): Error {
const reason = signal?.reason;
if (reason instanceof Error && reason.name === "AbortError") return reason;
const error = new Error("render aborted");
error.name = "AbortError";
return error;
}
/** "30000/1001" → 29.97; "0/0" → 0. */
export function parseFrameRate(value: string | undefined): number {
if (!value) return 0;
const [num, den] = value.split("/").map(Number);
if (num === undefined || !Number.isFinite(num)) return 0;
const rate = den === undefined ? num : den > 0 ? num / den : 0;
return Number.isFinite(rate) ? Math.round(rate * 1000) / 1000 : 0;
}
interface ProbeJson {
streams?: Array<{
codec_type?: string;
codec_name?: string;
width?: number;
height?: number;
r_frame_rate?: string;
tags?: Record<string, unknown>;
}>;
format?: { duration?: string; size?: string; tags?: Record<string, unknown> };
}
/** ffprobe JSON → report without loudness. Tags are flattened as `format.x` / `stream.N.x`. */
export function parseProbeReport(json: ProbeJson, fileSize: number): FfprobeReport {
const streams = json.streams ?? [];
const video = streams.find((stream) => stream.codec_type === "video");
const audio = streams.find((stream) => stream.codec_type === "audio");
const tags: Record<string, string> = {};
for (const [key, value] of Object.entries(json.format?.tags ?? {})) {
tags[`format.${key}`] = String(value);
}
streams.forEach((stream, index) => {
for (const [key, value] of Object.entries(stream.tags ?? {})) {
tags[`stream.${index}.${key}`] = String(value);
}
});
const seconds = Number(json.format?.duration);
const size = Number(json.format?.size);
return {
durationMs: Number.isFinite(seconds) && seconds > 0 ? Math.round(seconds * 1000) : 0,
width: video?.width ?? 0,
height: video?.height ?? 0,
fps: parseFrameRate(video?.r_frame_rate),
videoCodec: video?.codec_name ?? "none",
audioCodec: audio?.codec_name ?? null,
sizeBytes: Number.isFinite(size) && size > 0 ? Math.round(size) : fileSize,
integratedLufs: null,
truePeakDb: null,
tags,
};
}
/** Integrated loudness and true peak from the ebur128 summary on stderr. */
export function parseEbur128Summary(stderr: string): {
integratedLufs: number | null;
truePeakDb: number | null;
} {
const at = stderr.lastIndexOf("Summary:");
if (at < 0) return { integratedLufs: null, truePeakDb: null };
const summary = stderr.slice(at);
const integrated = /I:\s+(-?\d+(?:\.\d+)?) LUFS/.exec(summary);
const peak = /True peak:\s+Peak:\s+(-?\d+(?:\.\d+)?|-inf) dBFS/.exec(summary);
const lufs = integrated ? Number(integrated[1]) : null;
const tp = peak && peak[1] !== "-inf" ? Number(peak[1]) : null;
return {
// ebur128 reports -70 LUFS (its gate) for silence; that is "no programme".
integratedLufs: lufs !== null && Number.isFinite(lufs) && lufs > -70 ? lufs : null,
truePeakDb: tp !== null && Number.isFinite(tp) ? tp : null,
};
}
// ---------------------------------------------------------------------------
// Two-pass loudnorm
// ---------------------------------------------------------------------------
const LOUDNORM = /loudnorm=((?:[^,;[\]\\]|\\.)*)/g;
interface LoudnormSite {
/** Index of the options string inside the graph. */
start: number;
end: number;
options: string;
}
/** The single linear loudnorm without measured values, or null. */
export function findTwoPassLoudnorm(graph: string): LoudnormSite | null {
const matches = [...graph.matchAll(LOUDNORM)];
if (matches.length !== 1) return null;
const match = matches[0]!;
const options = match[1] ?? "";
const parts = options.split(":");
if (!parts.includes("linear=true")) return null;
if (parts.some((part) => part.startsWith("measured_"))) return null;
const start = (match.index ?? 0) + "loudnorm=".length;
return { start, end: start + options.length, options };
}
export interface LoudnormMeasurement {
inputI: number;
inputTp: number;
inputLra: number;
inputThresh: number;
targetOffset: number;
}
/** The JSON block loudnorm prints after its summary line; null when absent or not finite. */
export function parseLoudnormJson(stderr: string): LoudnormMeasurement | null {
const at = stderr.lastIndexOf("Parsed_loudnorm");
if (at < 0) return null;
const open = stderr.indexOf("{", at);
const close = stderr.indexOf("}", open);
if (open < 0 || close < 0) return null;
let parsed: Record<string, unknown>;
try {
parsed = JSON.parse(stderr.slice(open, close + 1)) as Record<string, unknown>;
} catch {
return null;
}
const read = (key: string) => {
const value = Number(parsed[key]);
return Number.isFinite(value) ? value : null;
};
const inputI = read("input_i");
const inputTp = read("input_tp");
const inputLra = read("input_lra");
const inputThresh = read("input_thresh");
const targetOffset = read("target_offset");
if (
inputI === null ||
inputTp === null ||
inputLra === null ||
inputThresh === null ||
targetOffset === null
) {
return null;
}
return { inputI, inputTp, inputLra, inputThresh, targetOffset };
}
function withLoudnormOptions(graph: string, site: LoudnormSite, extra: string): string {
return `${graph.slice(0, site.start)}${site.options}:${extra}${graph.slice(site.end)}`;
}
function measuredOptions(m: LoudnormMeasurement): string {
const f = (value: number) => value.toFixed(2);
return [
`measured_I=${f(m.inputI)}`,
`measured_TP=${f(m.inputTp)}`,
`measured_LRA=${f(m.inputLra)}`,
`measured_thresh=${f(m.inputThresh)}`,
`offset=${f(m.targetOffset)}`,
].join(":");
}
// ---------------------------------------------------------------------------
// Renderer
// ---------------------------------------------------------------------------
interface RunOptions {
signal?: AbortSignal;
/** Called with the output time reached, in ms. */
onTime?: (ms: number) => void;
}
export class FfmpegMediaRenderer implements MediaRenderer {
readonly id = "ffmpeg";
private readonly ffmpegPath: string;
private readonly ffprobePath: string;
private readonly vaapiDevice: string;
private readonly killGraceMs: number;
private encoders: Promise<string[]> | null = null;
constructor(private readonly options: FfmpegMediaRendererOptions = {}) {
this.ffmpegPath = options.ffmpegPath ?? "ffmpeg";
this.ffprobePath = options.ffprobePath ?? "ffprobe";
this.vaapiDevice = options.vaapiDevice ?? DEFAULT_VAAPI_DEVICE;
this.killGraceMs = options.killGraceMs ?? DEFAULT_KILL_GRACE_MS;
}
hardwareEncoders(): Promise<string[]> {
this.encoders ??= this.detectEncoders().catch((error) => {
getLogger().warn("media hardware encoder detection failed", {
reason: error instanceof Error ? error.message : String(error),
});
return [];
});
return this.encoders;
}
private async detectEncoders(): Promise<string[]> {
const { stdout } = await runMediaTool(this.ffmpegPath, ["-hide_banner", "-encoders"]);
const listed = new Set(
stdout
.toString("utf8")
.split("\n")
.map((line) => line.trim().split(/\s+/)[1])
.filter((name): name is string => Boolean(name)),
);
const hasVaapiDevice = await access(this.vaapiDevice).then(
() => true,
() => false,
);
const usable: string[] = [];
for (const encoder of HARDWARE_CANDIDATES) {
if (!listed.has(encoder)) continue;
const vaapi = encoder.endsWith("_vaapi");
if (vaapi && !hasVaapiDevice) continue;
// One-frame test encode: listed encoders often lack the device or driver.
const args = ["-hide_banner", "-v", "error", "-nostdin"];
if (vaapi) args.push("-vaapi_device", this.vaapiDevice);
args.push("-f", "lavfi", "-i", "color=c=black:s=256x256:r=30:d=0.1", "-frames:v", "1");
if (vaapi) args.push("-vf", "format=nv12,hwupload");
args.push("-c:v", encoder, "-f", "null", "-");
try {
await runMediaTool(this.ffmpegPath, args);
usable.push(encoder);
} catch {
// Not usable on this host.
}
}
return usable;
}
async render(
plan: RenderPlan,
options: { signal?: AbortSignal; onProgress?: (progress: MediaRenderProgress) => void } = {},
): Promise<void> {
const { signal, onProgress } = options;
if (signal?.aborted) throw abortError(signal);
const workDir = await this.workDir(plan.workDir);
const weight = (step: RenderStep) =>
step.kind === "ffmpeg" ? Math.max(step.durationMs, 1) : 0;
const total = plan.steps.reduce((sum, step) => sum + weight(step), 0) || 1;
let done = 0;
const report = (stepId: string, fraction: number, stepWeight: number) => {
const value = (done + Math.min(1, Math.max(0, fraction)) * stepWeight) / total;
onProgress?.({ stepId, progress: Math.min(1, Math.max(0, value)) });
};
const written: string[] = [];
try {
for (const step of plan.steps) {
if (signal?.aborted) throw abortError(signal);
switch (step.kind) {
case "write_file": {
const file = await this.insideWorkDir(workDir, step.path);
await mkdir(path.dirname(file), { recursive: true });
await writeFile(file, step.content, "utf8");
written.push(file);
break;
}
case "motion":
case "inpaint":
case "synthetic_identity": {
const ready = await access(step.outPath).then(
() => true,
() => false,
);
if (!ready) throw new Error(`${step.kind} step ${step.id} was not prepared`);
break;
}
case "ffmpeg": {
for (const produced of step.produces) await this.insideWorkDir(workDir, produced);
written.push(...step.produces);
await this.runStep(step, plan, workDir, {
signal,
report: (fraction) => report(step.id, fraction, weight(step)),
});
done += weight(step);
report(step.id, 0, 0);
break;
}
}
}
} catch (error) {
await Promise.all(written.map((file) => rm(file, { force: true }).catch(() => undefined)));
if (signal?.aborted) throw abortError(signal);
throw error;
}
}
private async runStep(
step: Extract<RenderStep, { kind: "ffmpeg" }>,
plan: RenderPlan,
workDir: string,
options: { signal?: AbortSignal; report: (fraction: number) => void },
): Promise<void> {
let argv = [...step.argv];
const vaapi = argv.some((arg, i) => argv[i - 1] === "-c:v" && arg.endsWith("_vaapi"));
const prefix = vaapi ? ["-vaapi_device", this.vaapiDevice] : [];
const durationMs = Math.max(step.durationMs, 1);
const twoPass = await this.loudnormPlan(argv, plan, workDir);
if (twoPass) {
let measurement: LoudnormMeasurement | null = null;
try {
const { stderr } = await this.run([...prefix, ...twoPass.measureArgv], {
signal: options.signal,
onTime: (ms) => options.report((ms / durationMs) * 0.5),
});
measurement = parseLoudnormJson(stderr);
} finally {
if (twoPass.measureScript) await rm(twoPass.measureScript, { force: true });
}
if (measurement) {
argv = await twoPass.apply(measuredOptions(measurement));
} else {
getLogger().warn("loudnorm measurement unreadable; rendering single pass", {
stepId: step.id,
});
}
}
const offset = twoPass ? 0.5 : 0;
const scale = twoPass ? 0.5 : 1;
await this.run([...prefix, ...argv], {
signal: options.signal,
onTime: (ms) => options.report(offset + (ms / durationMs) * scale),
});
options.report(1);
}
/**
* Measurement argv and a function that writes the measured filter into the
* render argv (or its filtergraph script), or null when the step needs no
* measurement pass.
*/
private async loudnormPlan(
argv: string[],
plan: RenderPlan,
workDir: string,
): Promise<{
measureArgv: string[];
measureScript: string | null;
apply: (extra: string) => Promise<string[]>;
} | null> {
const inline = argv.indexOf("-filter_complex");
const scriptAt = argv.indexOf("-filter_complex_script");
const flagAt = inline >= 0 ? inline : scriptAt;
if (flagAt < 0 || flagAt + 1 >= argv.length) return null;
let graph: string;
let scriptPath: string | null = null;
if (inline >= 0) {
graph = argv[inline + 1]!;
} else {
scriptPath = await this.insideWorkDir(workDir, argv[scriptAt + 1]!);
let content: string | null = null;
for (const step of plan.steps) {
if (step.kind !== "write_file") continue;
if ((await this.insideWorkDir(workDir, step.path)) === scriptPath) content = step.content;
}
if (content === null) return null;
graph = content;
}
const site = findTwoPassLoudnorm(graph);
if (!site) return null;
const tail = argv.slice(flagAt + 2);
const maps: string[] = [];
let duration: string | null = null;
for (let i = 0; i < tail.length - 1; i++) {
if (tail[i] === "-map") maps.push("-map", tail[i + 1]!);
if (tail[i] === "-t") duration = tail[i + 1]!;
}
const measureGraph = withLoudnormOptions(graph, site, "print_format=json");
let measureScript: string | null = null;
let graphArg = measureGraph;
if (scriptPath) {
measureScript = path.join(path.dirname(scriptPath), "filtergraph.measure.txt");
await writeFile(measureScript, measureGraph, "utf8");
graphArg = measureScript;
}
const measureArgv = [
...argv.slice(0, flagAt + 1),
graphArg,
...maps,
...(duration ? ["-t", duration] : []),
"-f",
"null",
"-",
];
return {
measureArgv,
measureScript,
apply: async (extra) => {
const next = withLoudnormOptions(graph, site, extra);
const out = [...argv];
if (scriptPath) await writeFile(scriptPath, next, "utf8");
else out[flagAt + 1] = next;
return out;
},
};
}
/** Spawn ffmpeg with progress on stdout. Rejects with the stderr tail on failure. */
private run(args: string[], options: RunOptions): Promise<{ stderr: string }> {
const { signal } = options;
if (signal?.aborted) return Promise.reject(abortError(signal));
return new Promise((resolve, reject) => {
const child = spawn(this.ffmpegPath, ["-progress", "pipe:1", "-nostats", ...args], {
stdio: ["ignore", "pipe", "pipe"],
shell: false,
windowsHide: true,
});
let stderr = "";
let pending = "";
let killTimer: NodeJS.Timeout | null = null;
const onAbort = () => {
child.kill("SIGTERM");
killTimer = setTimeout(() => child.kill("SIGKILL"), this.killGraceMs);
killTimer.unref();
};
signal?.addEventListener("abort", onAbort, { once: true });
child.stdout.setEncoding("utf8");
child.stdout.on("data", (chunk: string) => {
pending += chunk;
const lines = pending.split("\n");
pending = lines.pop() ?? "";
for (const line of lines) {
const match = /^out_time_(us|ms)=(\d+)/.exec(line.trim());
// Both keys are microseconds in ffmpeg 5+ (out_time_ms is misnamed).
if (match) options.onTime?.(Number(match[2]) / 1000);
}
});
child.stderr.setEncoding("utf8");
child.stderr.on("data", (chunk: string) => {
stderr += chunk;
if (stderr.length > STDERR_TAIL_BYTES * 2) stderr = stderr.slice(-STDERR_TAIL_BYTES);
});
child.on("error", (error: NodeJS.ErrnoException) => {
signal?.removeEventListener("abort", onAbort);
reject(error.code === "ENOENT" ? new MediaToolMissingError(this.ffmpegPath) : error);
});
child.on("close", (code) => {
signal?.removeEventListener("abort", onAbort);
if (killTimer) clearTimeout(killTimer);
if (signal?.aborted) {
reject(abortError(signal));
return;
}
if (code !== 0) {
const tail = stderr.trim().split("\n").slice(-4).join(" ");
reject(new Error(`ffmpeg failed (${code}): ${tail}`));
return;
}
resolve({ stderr });
});
});
}
async probe(
file: string,
options: { measureLoudness?: boolean; signal?: AbortSignal } = {},
): Promise<FfprobeReport> {
const target = await this.checked(file);
const { stdout } = await runMediaTool(
this.ffprobePath,
["-v", "error", "-print_format", "json", "-show_format", "-show_streams", "--", target],
{ signal: options.signal },
);
const report = parseProbeReport(
JSON.parse(stdout.toString("utf8")) as ProbeJson,
(await stat(target)).size,
);
if (!options.measureLoudness || report.audioCodec === null) return report;
const { stderr } = await runMediaTool(
this.ffmpegPath,
[
"-hide_banner",
"-nostats",
"-nostdin",
"-i",
target,
"-map",
"0:a:0",
"-af",
"ebur128=peak=true",
"-f",
"null",
"-",
],
{ signal: options.signal },
);
return { ...report, ...parseEbur128Summary(stderr) };
}
async extractFrame(input: {
path: string;
tMs: number;
outPath: string;
width?: number;
signal?: AbortSignal;
}): Promise<void> {
const source = await this.checked(input.path);
if (path.extname(input.outPath).toLowerCase() !== ".png") {
throw new Error("frames are extracted as PNG");
}
const outPath = await this.checked(input.outPath);
if (!Number.isFinite(input.tMs) || input.tMs < 0) throw new Error("invalid frame time");
const args = [
"-hide_banner",
"-v",
"error",
"-nostdin",
"-y",
"-ss",
(input.tMs / 1000).toFixed(3),
"-i",
source,
"-frames:v",
"1",
];
if (input.width !== undefined) {
const width = Math.max(2, Math.floor(input.width / 2) * 2);
args.push("-vf", `scale=${width}:-2`);
}
args.push(
"-map_metadata",
"-1",
"-fflags",
"+bitexact",
"-flags:v",
"+bitexact",
"-f",
"image2",
"-c:v",
"png",
outPath,
);
await runMediaTool(this.ffmpegPath, args, { signal: input.signal });
const written = await stat(outPath).catch(() => null);
if (!written || written.size === 0) throw new Error("no frame at the requested time");
}
private async checked(file: string): Promise<string> {
if (!path.isAbsolute(file)) throw new Error("media paths must be absolute");
return this.options.root ? assertInside(this.options.root, file) : path.resolve(file);
}
private async workDir(dir: string): Promise<string> {
const resolved = await this.checked(dir);
await mkdir(resolved, { recursive: true });
return resolved;
}
/** Steps may only write inside the plan's work directory. */
private async insideWorkDir(workDir: string, file: string): Promise<string> {
if (!path.isAbsolute(file)) throw new Error("plan paths must be absolute");
return assertInside(workDir, file);
}
}

View File

@ -0,0 +1,194 @@
import { execFile } from "node:child_process";
/*
* Local ffmpeg/ffprobe helpers shared by the folder source and the media
* emulator. Every call is an argv array without a shell; nothing here builds a
* command string.
*/
export class MediaToolMissingError extends Error {
constructor(readonly tool: string) {
super(`${tool} is not installed or not on PATH; install ffmpeg to use local media processing`);
this.name = "MediaToolMissingError";
}
}
export interface MediaToolPaths {
ffmpegPath?: string;
ffprobePath?: string;
}
/** Run a media tool and collect stdout (binary) and stderr. Rejects on non-zero exit. */
export function runMediaTool(
bin: string,
args: string[],
options: { signal?: AbortSignal; maxBuffer?: number } = {},
): Promise<{ stdout: Buffer; stderr: string }> {
options.signal?.throwIfAborted();
return new Promise((resolve, reject) => {
execFile(
bin,
args,
{
encoding: "buffer",
maxBuffer: options.maxBuffer ?? 64 * 1024 * 1024,
signal: options.signal,
shell: false,
windowsHide: true,
},
(error, stdout, stderr) => {
if (options.signal?.aborted) {
reject(options.signal.reason);
return;
}
if (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
reject(new MediaToolMissingError(bin));
return;
}
const tail = stderr.toString("utf8").trim().split("\n").slice(-3).join(" ");
reject(new Error(`${bin} failed: ${tail || error.message}`));
return;
}
resolve({ stdout, stderr: stderr.toString("utf8") });
},
);
});
}
export interface ProbedMedia {
durationMs: number | null;
width: number | null;
height: number | null;
fps: number | null;
codec: string | null;
rotation: number;
hasAudio: boolean;
hasVideo: boolean;
capturedAt: Date | null;
gpsLat: number | null;
gpsLon: number | null;
deviceModel: string | null;
}
interface FfprobeStream {
codec_type?: string;
codec_name?: string;
width?: number;
height?: number;
avg_frame_rate?: string;
r_frame_rate?: string;
duration?: string;
tags?: Record<string, string>;
side_data_list?: Array<{ rotation?: number }>;
}
interface FfprobeOutput {
streams?: FfprobeStream[];
format?: { duration?: string; tags?: Record<string, string> };
frames?: Array<{ tags?: Record<string, string> }>;
}
/** Probe a local file. Throws when ffprobe is missing or cannot read the file. */
export async function probeMedia(
file: string,
options: { ffprobePath?: string; signal?: AbortSignal; frameTags?: boolean } = {},
): Promise<ProbedMedia> {
const args = ["-v", "error", "-print_format", "json", "-show_format", "-show_streams"];
if (options.frameTags) args.push("-show_frames", "-read_intervals", "%+#1");
args.push("--", file);
const { stdout } = await runMediaTool(options.ffprobePath ?? "ffprobe", args, {
signal: options.signal,
});
return parseFfprobe(JSON.parse(stdout.toString("utf8")) as FfprobeOutput);
}
export function parseFfprobe(output: FfprobeOutput): ProbedMedia {
const streams = output.streams ?? [];
const video = streams.find((stream) => stream.codec_type === "video");
const tags = collectTags(output);
const rotation = video ? streamRotation(video) : 0;
const quarterTurn = Math.abs(rotation) % 180 === 90;
const durationSeconds = Number(output.format?.duration ?? video?.duration);
const gps = parseIso6709(
tags.get("com.apple.quicktime.location.iso6709") ?? tags.get("location"),
);
return {
durationMs:
Number.isFinite(durationSeconds) && durationSeconds > 0
? Math.round(durationSeconds * 1000)
: null,
width: (quarterTurn ? video?.height : video?.width) ?? null,
height: (quarterTurn ? video?.width : video?.height) ?? null,
fps: video ? (parseRate(video.avg_frame_rate) ?? parseRate(video.r_frame_rate)) : null,
codec: video?.codec_name ?? null,
rotation,
hasAudio: streams.some((stream) => stream.codec_type === "audio"),
hasVideo: Boolean(video),
capturedAt: parseCaptureDate(
tags.get("com.apple.quicktime.creationdate") ??
tags.get("datetimeoriginal") ??
tags.get("creation_time") ??
tags.get("datetime"),
),
gpsLat: gps?.lat ?? null,
gpsLon: gps?.lon ?? null,
deviceModel: tags.get("com.apple.quicktime.model") ?? tags.get("model") ?? null,
};
}
/** Format, stream and first-frame tags merged with lowercase keys; format wins. */
function collectTags(output: FfprobeOutput): Map<string, string> {
const merged = new Map<string, string>();
const sources = [
output.format?.tags,
...(output.streams ?? []).map((stream) => stream.tags),
...(output.frames ?? []).map((frame) => frame.tags),
];
for (const source of sources) {
for (const [key, value] of Object.entries(source ?? {})) {
const normalized = key.toLowerCase();
if (!merged.has(normalized) && typeof value === "string" && value.trim())
merged.set(normalized, value.trim());
}
}
return merged;
}
function streamRotation(stream: FfprobeStream): number {
const side = stream.side_data_list?.find((entry) => typeof entry.rotation === "number");
if (side?.rotation !== undefined) return side.rotation;
const tagged = Number(stream.tags?.rotate);
return Number.isFinite(tagged) ? tagged : 0;
}
function parseRate(rate: string | undefined): number | null {
if (!rate) return null;
const [num, den] = rate.split("/").map(Number);
if (!num || !Number.isFinite(num)) return null;
const value = den ? num / den : num;
return Number.isFinite(value) && value > 0 ? Math.round(value * 1000) / 1000 : null;
}
/** ISO 8601 or EXIF ("2026:05:01 09:00:00") timestamps; null when unparseable. */
export function parseCaptureDate(value: string | undefined): Date | null {
if (!value) return null;
const exif = /^(\d{4}):(\d{2}):(\d{2})[ T](\d{2}):(\d{2}):(\d{2})/.exec(value);
const date = exif
? new Date(`${exif[1]}-${exif[2]}-${exif[3]}T${exif[4]}:${exif[5]}:${exif[6]}Z`)
: new Date(value);
const time = date.getTime();
return Number.isFinite(time) && time > 0 ? date : null;
}
/** "+52.5200+013.4050+034.000/" → { lat, lon }; null when absent or out of range. */
export function parseIso6709(value: string | undefined): { lat: number; lon: number } | null {
if (!value) return null;
const match = /^([+-]\d+(?:\.\d+)?)([+-]\d+(?:\.\d+)?)/.exec(value.trim());
if (!match) return null;
const lat = Number(match[1]);
const lon = Number(match[2]);
if (!Number.isFinite(lat) || !Number.isFinite(lon)) return null;
if (Math.abs(lat) > 90 || Math.abs(lon) > 180) return null;
return { lat, lon };
}

View File

@ -0,0 +1,177 @@
import { mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { MediaSourceItem } from "@rakazo/adapter-kit";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { ffmpegAvailable } from "./media-emulator-clips.js";
import { FolderMediaSourceProvider } from "./media-folder-source.js";
import { parseIso6709, runMediaTool } from "./media-folder-source-probe.js";
let root: string;
const provider = new FolderMediaSourceProvider();
async function touch(relative: string, content = "") {
const file = path.join(root, relative);
await mkdir(path.dirname(file), { recursive: true });
await writeFile(file, content);
}
async function listAll(limit: number, cursor: string | null = null) {
const pages: MediaSourceItem[][] = [];
let next = cursor;
do {
const page = await provider.list({ config: { path: root }, cursor: next, limit });
expect(page.items.length).toBeLessThanOrEqual(limit);
pages.push(page.items);
next = page.nextCursor;
} while (next);
return pages.flat().map((item) => item.externalId);
}
beforeEach(async () => {
root = await mkdtemp(path.join(tmpdir(), "media-folder-"));
});
afterEach(async () => {
await rm(root, { recursive: true, force: true });
});
describe("FolderMediaSourceProvider", () => {
it("lists media in stable code-unit order across pages", async () => {
for (const name of ["b/c.mp4", "b-d.jpg", "a.MOV", "b/a/z.webm", "c.png", "notes.txt"])
await touch(name);
const ids = await listAll(2);
expect(ids).toEqual(["a.MOV", "b-d.jpg", "b/a/z.webm", "b/c.mp4", "c.png"]);
});
it("keeps the cursor stable when files are added mid-sync", async () => {
for (const name of ["a.mp4", "c.mp4", "e.mp4", "g.mp4"]) await touch(name);
const first = await provider.list({ config: { path: root }, cursor: null, limit: 2 });
expect(first.items.map((item) => item.externalId)).toEqual(["a.mp4", "c.mp4"]);
await touch("b.mp4"); // before the cursor: picked up by the next full sync
await touch("d.mp4"); // after the cursor: part of this sync
const rest = await listAll(2, first.nextCursor);
expect(rest).toEqual(["d.mp4", "e.mp4", "g.mp4"]);
expect(await listAll(10)).toEqual(["a.mp4", "b.mp4", "c.mp4", "d.mp4", "e.mp4", "g.mp4"]);
});
it("skips hidden entries and symlinks without following them", async () => {
await touch("keep/clip.mp4");
await touch(".hidden.mp4");
await touch(".cache/inside.mp4");
await touch("keep/.DS_Store.jpg");
const outside = await mkdtemp(path.join(tmpdir(), "media-folder-outside-"));
try {
await writeFile(path.join(outside, "secret.mp4"), "");
await symlink(path.join(outside, "secret.mp4"), path.join(root, "link.mp4"));
await symlink(outside, path.join(root, "linked-dir"));
await symlink(path.join(root, "keep"), path.join(root, "keep-alias"));
expect(await listAll(10)).toEqual(["keep/clip.mp4"]);
await expect(provider.resolve({ path: root }, "link.mp4")).rejects.toThrow();
await expect(provider.resolve({ path: root }, "linked-dir/secret.mp4")).rejects.toThrow();
await expect(provider.resolve({ path: root }, "keep-alias/clip.mp4")).rejects.toThrow();
} finally {
await rm(outside, { recursive: true, force: true });
}
});
it("resolves inside the root and rejects traversal", async () => {
await touch("keep/clip.mp4");
await touch(".hidden.mp4");
await expect(provider.resolve({ path: root }, "keep/clip.mp4")).resolves.toBe(
path.join(root, "keep/clip.mp4"),
);
for (const bad of [
"../x",
"../x.mp4",
"keep/../../x.mp4",
"/etc/passwd",
".hidden.mp4",
"keep",
])
await expect(provider.resolve({ path: root }, bad), bad).rejects.toThrow();
});
it("rejects relative roots and forged cursors", async () => {
await expect(
provider.list({ config: { path: "media" }, cursor: null, limit: 5 }),
).rejects.toThrow();
await expect(
provider.list({ config: { path: root }, cursor: "not base64url!", limit: 5 }),
).rejects.toThrow();
});
it("pages a 1,000-file folder without duplicates or gaps", async () => {
const expected: string[] = [];
for (let dir = 0; dir < 10; dir++) {
for (let file = 0; file < 100; file++) {
const relative = `d${dir}/f${String(file).padStart(3, "0")}.${file % 3 === 0 ? "jpg" : "mp4"}`;
// Unreadable media must not abort a page; most files are empty and skip probing.
await touch(relative, file % 50 === 7 ? "garbage, not a video" : "");
expected.push(relative);
}
}
await touch("d3/ignore.txt");
const pages: MediaSourceItem[][] = [];
let cursor: string | null = null;
do {
const page = await provider.list({ config: { path: root }, cursor, limit: 137 });
pages.push(page.items);
cursor = page.nextCursor;
} while (cursor);
const items = pages.flat();
expect(pages.length).toBe(Math.ceil(1000 / 137));
expect(items.map((item) => item.externalId)).toEqual(expected.sort());
expect(new Set(items.map((item) => item.externalId)).size).toBe(1000);
const garbage = items.find((item) => item.externalId === "d0/f007.mp4");
expect(garbage).toMatchObject({ durationMs: null, width: null, sizeBytes: 20, kind: "video" });
expect(garbage?.capturedAt).toBeInstanceOf(Date);
expect(garbage?.revision).toMatch(/^\d+-20$/);
expect(items.find((item) => item.externalId === "d0/f000.jpg")?.kind).toBe("photo");
});
it.skipIf(!ffmpegAvailable())(
"reads duration, size, capture time and GPS with ffprobe",
async () => {
await mkdir(path.join(root, "trip"), { recursive: true });
await runMediaTool("ffmpeg", [
"-v",
"error",
"-f",
"lavfi",
"-i",
"testsrc2=s=320x180:r=25:d=1",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-metadata",
"creation_time=2026-05-02T08:00:00Z",
"-metadata",
"location=+52.5200+013.4050/",
path.join(root, "trip/clip.mp4"),
]);
const [item] = (await provider.list({ config: { path: root }, cursor: null, limit: 5 }))
.items;
expect(item).toMatchObject({
externalId: "trip/clip.mp4",
kind: "video",
durationMs: 1000,
width: 320,
height: 180,
fps: 25,
gpsLat: 52.52,
gpsLon: 13.405,
albumNames: ["trip"],
});
expect(item?.capturedAt?.toISOString()).toBe("2026-05-02T08:00:00.000Z");
},
);
it("parses ISO 6709 locations", () => {
expect(parseIso6709("+52.5200+013.4050+034.000/")).toEqual({ lat: 52.52, lon: 13.405 });
expect(parseIso6709("-33.8688+151.2093/")).toEqual({ lat: -33.8688, lon: 151.2093 });
expect(parseIso6709("+95.0+013.0/")).toBeNull();
expect(parseIso6709("nowhere")).toBeNull();
});
});

View File

@ -0,0 +1,221 @@
import type { Dirent } from "node:fs";
import { lstat, readdir } from "node:fs/promises";
import path from "node:path";
import type {
MediaSourceItem,
MediaSourceListRequest,
MediaSourceListResult,
MediaSourceProvider,
} from "@rakazo/adapter-kit";
import type { MediaAssetKind } from "@rakazo/contracts";
import { PathOutsideRootError, resolveInside } from "@rakazo/media/node";
import { z } from "zod";
import { probeMedia } from "./media-folder-source-probe.js";
/*
* Server-side folder source. The deployment owner picks an absolute root; the
* walk lists media files below it in a stable order (relative POSIX paths,
* code-unit sorted) so the opaque cursor (the last relative path) stays valid
* while files are added. Hidden entries and symlinks are skipped and never
* followed. Only the directories needed for one page are read.
*/
export const FOLDER_MEDIA_EXTENSIONS: Readonly<Record<string, MediaAssetKind>> = {
mp4: "video",
mov: "video",
m4v: "video",
mkv: "video",
webm: "video",
avi: "video",
jpg: "photo",
jpeg: "photo",
png: "photo",
heic: "photo",
heif: "photo",
webp: "photo",
};
const FolderConfigSchema = z.object({
path: z.string().min(1).refine(path.isAbsolute, "folder path must be absolute"),
});
const MAX_PAGE = 1000;
const PROBE_CONCURRENCY = 4;
export interface FolderMediaSourceOptions {
ffprobePath?: string;
}
export function encodeFolderCursor(relativePath: string): string {
return Buffer.from(relativePath, "utf8").toString("base64url");
}
export function decodeFolderCursor(cursor: string | null): string | null {
if (!cursor) return null;
const decoded = Buffer.from(cursor, "base64url").toString("utf8");
if (!decoded || decoded.includes("\0") || encodeFolderCursor(decoded) !== cursor)
throw new Error("invalid folder cursor");
return decoded;
}
function mediaKindOf(name: string): MediaAssetKind | null {
const ext = path.extname(name).slice(1).toLowerCase();
return FOLDER_MEDIA_EXTENSIONS[ext] ?? null;
}
/** Sort key that makes a depth-first walk yield relative paths in code-unit order. */
function entryKey(entry: Dirent): string {
return entry.isDirectory() ? `${entry.name}/` : entry.name;
}
function compareKeys(a: string, b: string): number {
return a < b ? -1 : a > b ? 1 : 0;
}
/**
* Yield relative paths of media files strictly after `after`, in order.
* Directories whose whole subtree sorts before the cursor are not read.
*/
async function* walkAfter(
root: string,
relativeDir: string,
after: string | null,
signal?: AbortSignal,
): AsyncGenerator<string> {
signal?.throwIfAborted();
let entries: Dirent[];
try {
entries = await readdir(path.join(root, relativeDir), { withFileTypes: true });
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
// A directory removed or locked mid-walk is skipped; the root itself must exist.
if (relativeDir && (code === "ENOENT" || code === "EACCES" || code === "ENOTDIR")) return;
throw error;
}
const visible = entries
.filter((entry) => !entry.name.startsWith(".") && !entry.isSymbolicLink())
.filter((entry) => entry.isDirectory() || (entry.isFile() && mediaKindOf(entry.name)))
.map((entry) => ({ entry, key: entryKey(entry) }))
.sort((a, b) => compareKeys(a.key, b.key));
for (const { entry, key } of visible) {
const relative = relativeDir ? `${relativeDir}/${key}` : key;
if (entry.isDirectory()) {
// Subtree entirely before the cursor: every path in it starts with `relative`.
if (after !== null && relative < after && !after.startsWith(relative)) continue;
yield* walkAfter(root, relative.slice(0, -1), after, signal);
} else if (after === null || relative > after) {
yield relative;
}
}
}
async function mapLimited<T, R>(
items: T[],
limit: number,
fn: (item: T) => Promise<R>,
): Promise<R[]> {
const out = new Array<R>(items.length);
let next = 0;
const workers = Array.from({ length: Math.min(limit, items.length) }, async () => {
while (next < items.length) {
const index = next++;
out[index] = await fn(items[index] as T);
}
});
await Promise.all(workers);
return out;
}
export class FolderMediaSourceProvider implements MediaSourceProvider {
readonly kind = "folder" as const;
constructor(private readonly options: FolderMediaSourceOptions = {}) {}
async list(request: MediaSourceListRequest): Promise<MediaSourceListResult> {
const { path: root } = FolderConfigSchema.parse(request.config);
const limit = Math.max(1, Math.min(MAX_PAGE, Math.floor(request.limit)));
const after = decodeFolderCursor(request.cursor);
const relatives: string[] = [];
let more = false;
for await (const relative of walkAfter(root, "", after, request.signal)) {
if (relatives.length === limit) {
more = true;
break;
}
relatives.push(relative);
}
const items = await mapLimited(relatives, PROBE_CONCURRENCY, (relative) =>
this.describe(root, relative, request.signal),
);
const present = items.filter((item): item is MediaSourceItem => item !== null);
const last = relatives.at(-1);
return { items: present, nextCursor: more && last ? encodeFolderCursor(last) : null };
}
async resolve(config: Record<string, unknown>, externalId: string): Promise<string> {
const { path: root } = FolderConfigSchema.parse(config);
const segments = externalId.split("/");
if (segments.some((segment) => segment.startsWith(".")) || !mediaKindOf(externalId))
throw new PathOutsideRootError(externalId);
const absolute = await resolveInside(root, externalId);
// Symlinks are never followed, at any depth below the root.
let current = await resolveInside(root, ".");
for (const segment of segments) {
current = path.join(current, segment);
if ((await lstat(current)).isSymbolicLink()) throw new PathOutsideRootError(externalId);
}
if (!(await lstat(absolute)).isFile()) throw new PathOutsideRootError(externalId);
return absolute;
}
/**
* Stat and probe one file. Probe failures yield null metadata, never a failed
* page. The parent folder name doubles as an album for rule-based labels.
*/
private async describe(
root: string,
relative: string,
signal?: AbortSignal,
): Promise<MediaSourceItem | null> {
const absolute = path.join(root, relative);
let stats: Awaited<ReturnType<typeof lstat>>;
try {
stats = await lstat(absolute);
} catch {
return null; // Removed between walk and stat.
}
if (!stats.isFile()) return null;
const kind = mediaKindOf(relative) ?? "video";
let probed: Awaited<ReturnType<typeof probeMedia>> | null = null;
if (stats.size > 0) {
try {
probed = await probeMedia(absolute, {
ffprobePath: this.options.ffprobePath,
signal,
frameTags: kind === "photo",
});
} catch {
// Unreadable, unsupported, or no ffprobe on this host: keep the file with null metadata.
signal?.throwIfAborted();
probed = null;
}
}
return {
externalId: relative,
kind,
capturedAt: probed?.capturedAt ?? new Date(Math.trunc(stats.mtimeMs)),
durationMs: kind === "video" ? (probed?.durationMs ?? null) : null,
width: probed?.width ?? null,
height: probed?.height ?? null,
fps: kind === "video" ? (probed?.fps ?? null) : null,
gpsLat: probed?.gpsLat ?? null,
gpsLon: probed?.gpsLon ?? null,
deviceModel: probed?.deviceModel ?? null,
albumNames: relative.includes("/") ? [path.posix.basename(path.posix.dirname(relative))] : [],
isFavorite: false,
fileName: path.posix.basename(relative),
sizeBytes: stats.size,
revision: `${Math.trunc(stats.mtimeMs)}-${stats.size}`,
};
}
}

View File

@ -0,0 +1,10 @@
export type {
MotionFrameContext,
MotionTemplate,
ResvgMotionGraphicsRendererOptions,
} from "./media-motion/index.js";
export {
MOTION_TEMPLATES,
MotionTemplateError,
ResvgMotionGraphicsRenderer,
} from "./media-motion/index.js";

View File

@ -0,0 +1,56 @@
/** Easing and timing helpers shared by every motion template. All pure. */
export function clamp(value: number, min = 0, max = 1): number {
return Math.min(max, Math.max(min, value));
}
export function lerp(from: number, to: number, amount: number): number {
return from + (to - from) * amount;
}
/** Linear progress 0..1 of `t` through the window [start, end]. */
export function progress(t: number, start: number, end: number): number {
if (end <= start) return t >= end ? 1 : 0;
return clamp((t - start) / (end - start));
}
export function easeInCubic(x: number): number {
return x * x * x;
}
export function easeOutCubic(x: number): number {
return 1 - (1 - x) ** 3;
}
export function easeInOutCubic(x: number): number {
return x < 0.5 ? 4 * x * x * x : 1 - (-2 * x + 2) ** 3 / 2;
}
export function easeOutExpo(x: number): number {
return x >= 1 ? 1 : 1 - 2 ** (-10 * x);
}
/** Overshoots slightly before settling; for pop-in reveals. */
export function easeOutBack(x: number): number {
const c1 = 1.70158;
const c3 = c1 + 1;
return 1 + c3 * (x - 1) ** 3 + c1 * (x - 1) ** 2;
}
/**
* Opacity envelope for a clip of `durationMs`: fades in over `inMs` and out
* over the last `outMs`.
*/
export function fadeEnvelope(t: number, durationMs: number, inMs: number, outMs: number): number {
const fadeIn = inMs > 0 ? easeOutCubic(progress(t, 0, inMs)) : 1;
const fadeOut = outMs > 0 ? 1 - easeInCubic(progress(t, durationMs - outMs, durationMs)) : 1;
return clamp(Math.min(fadeIn, fadeOut));
}
/** Deterministic pseudo-random value in [0, 1) for integer seeds. */
export function hash01(a: number, b = 0): number {
let h = Math.imul(a ^ 0x9e3779b9, 0x85ebca6b) ^ Math.imul(b + 0x632be5ab, 0xc2b2ae35);
h = Math.imul(h ^ (h >>> 15), 0x2c1b3c6d);
h = Math.imul(h ^ (h >>> 12), 0x297a2d39);
return ((h ^ (h >>> 15)) >>> 0) / 4294967296;
}

View File

@ -0,0 +1,308 @@
/*
* Motion graphics renderer for Media Studio (intros, outros, lower thirds,
* caption previews, end cards).
*
* Technology decision: no Remotion and no headless Chromium. Remotion needs a
* paid company license above a team size and Chromium is a heavy, hard-to-
* sandbox runtime. Instead every template is a pure TypeScript function that
* returns an SVG document for a point in time. Frames are rasterized with
* @resvg/resvg-js (MPL-2.0; resvg itself is MPL-2.0, used unmodified as a
* library, commercially usable) and piped as PNGs into ffmpeg (LGPL/GPL
* depending on the build, invoked as a separate process with an argv array),
* which encodes a QuickTime Animation (qtrle, argb) MOV with alpha. Nothing
* here needs a network or a hosted service.
*/
import type { ChildProcess } from "node:child_process";
import { spawn } from "node:child_process";
import { randomUUID } from "node:crypto";
import { once } from "node:events";
import { readFile, rename, rm, stat } from "node:fs/promises";
import path from "node:path";
import type { MotionGraphicsRenderer, MotionRenderRequest } from "@rakazo/adapter-kit";
import type { BrandKit, MotionTemplateInfo, MotionTemplateRef } from "@rakazo/contracts";
import { BrandKitSchema, MotionTemplateRefSchema } from "@rakazo/contracts";
import { assertInside } from "@rakazo/media/node";
import type { RenderedImage, ResvgRenderOptions } from "@resvg/resvg-js";
import { renderAsync } from "@resvg/resvg-js";
import type { MotionFrameContext, MotionTemplate } from "./template.js";
import { brandFonts, MotionTemplateError, validateDuration, validateParams } from "./template.js";
import { captionBoldSocial, captionClean, captionKaraoke } from "./templates/captions.js";
import { endCardCta, endCardMinimal } from "./templates/end-cards.js";
import {
introElegantFade,
introGlitch,
introKineticType,
introLogoReveal,
} from "./templates/intros.js";
import { lowerThirdBar, lowerThirdClean, lowerThirdPill } from "./templates/lower-thirds.js";
import { outroLogo, outroSubscribe } from "./templates/outros.js";
export type { MotionFrameContext, MotionTemplate } from "./template.js";
export { MotionTemplateError } from "./template.js";
export const MOTION_TEMPLATES: readonly MotionTemplate[] = [
introLogoReveal,
introKineticType,
introGlitch,
introElegantFade,
outroLogo,
outroSubscribe,
lowerThirdClean,
lowerThirdBar,
lowerThirdPill,
captionClean,
captionKaraoke,
captionBoldSocial,
endCardCta,
endCardMinimal,
];
export function motionTemplate(id: string): MotionTemplate {
const template = MOTION_TEMPLATES.find((entry) => entry.info.id === id);
if (!template) throw new MotionTemplateError("unknown_template", `unknown template "${id}"`);
return template;
}
export const MIN_FRAME_SIDE = 16;
export const MAX_FRAME_SIDE = 3840;
export const MAX_FPS = 60;
/** Default cap on width × height × frames: ten seconds of 1080p60. */
export const DEFAULT_MAX_PIXEL_FRAMES = 1920 * 1080 * 60 * 10;
const MAX_LOGO_BYTES = 5 * 1024 * 1024;
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
const FONT_EXTENSIONS = new Set([".ttf", ".otf", ".ttc"]);
/**
* Read a logo as a data URI. Only PNG and SVG are accepted, and an SVG may not
* declare entities or reference anything outside itself.
*/
export async function loadLogoDataUri(file: string): Promise<string> {
const extension = path.extname(file).toLowerCase();
if (extension !== ".png" && extension !== ".svg")
throw new MotionTemplateError("invalid_asset", "logos must be PNG or SVG");
const info = await stat(file);
if (!info.isFile() || info.size > MAX_LOGO_BYTES)
throw new MotionTemplateError("invalid_asset", "logo is not a file or is too large");
const bytes = await readFile(file);
if (extension === ".png") {
if (!bytes.subarray(0, 8).equals(PNG_SIGNATURE))
throw new MotionTemplateError("invalid_asset", "logo is not a PNG");
return `data:image/png;base64,${bytes.toString("base64")}`;
}
const text = bytes.toString("utf8");
if (!/<svg[\s>]/i.test(text))
throw new MotionTemplateError("invalid_asset", "logo is not an SVG");
if (/<!DOCTYPE|<!ENTITY/i.test(text) || /\bhref\s*=\s*["']\s*(?!#|data:)/i.test(text))
throw new MotionTemplateError("invalid_asset", "SVG logos must be self-contained");
return `data:image/svg+xml;base64,${bytes.toString("base64")}`;
}
/** Validate a template reference and assemble the pure frame context. */
export function buildFrameContext(input: {
template: MotionTemplateRef;
kit: BrandKit;
width: number;
height: number;
logos: string[];
}): { template: MotionTemplate; ctx: MotionFrameContext } {
const ref = MotionTemplateRefSchema.parse(input.template);
const kit = BrandKitSchema.parse(input.kit);
const template = motionTemplate(ref.templateId);
validateDuration(template.info, ref.durationMs);
for (const side of [input.width, input.height]) {
if (!Number.isInteger(side) || side < MIN_FRAME_SIDE || side > MAX_FRAME_SIDE)
throw new MotionTemplateError(
"invalid_size",
`frame sides must be ${MIN_FRAME_SIDE}..${MAX_FRAME_SIDE} px`,
);
}
return {
template,
ctx: {
width: input.width,
height: input.height,
durationMs: ref.durationMs,
kit,
params: validateParams(template.info, ref.params),
logos: input.logos,
fonts: brandFonts(kit),
},
};
}
export interface RasterizeOptions {
fontFiles: string[];
defaultFontFamily?: string | null;
/** Fall back to system fonts for missing families and glyphs. */
loadSystemFonts: boolean;
}
export function rasterizeFrame(svg: string, options: RasterizeOptions): Promise<RenderedImage> {
const resvgOptions: ResvgRenderOptions = {
font: {
loadSystemFonts: options.loadSystemFonts,
fontFiles: options.fontFiles,
...(options.defaultFontFamily ? { defaultFontFamily: options.defaultFontFamily } : {}),
},
logLevel: "off",
};
return renderAsync(svg, resvgOptions);
}
export function frameCount(durationMs: number, fps: number): number {
return Math.max(1, Math.round((durationMs * fps) / 1000));
}
function abortError(): Error {
const error = new Error("motion render aborted");
error.name = "AbortError";
return error;
}
export interface ResvgMotionGraphicsRendererOptions {
ffmpegPath?: string;
/** When set, every asset path and the output path must lie inside it. */
root?: string;
maxPixelFrames?: number;
}
export class ResvgMotionGraphicsRenderer implements MotionGraphicsRenderer {
readonly id = "resvg";
private readonly ffmpegPath: string;
private readonly maxPixelFrames: number;
constructor(private readonly options: ResvgMotionGraphicsRendererOptions = {}) {
this.ffmpegPath = options.ffmpegPath ?? "ffmpeg";
this.maxPixelFrames = options.maxPixelFrames ?? DEFAULT_MAX_PIXEL_FRAMES;
}
templates(): MotionTemplateInfo[] {
return MOTION_TEMPLATES.map((template) => structuredClone(template.info));
}
async render(request: MotionRenderRequest): Promise<{ path: string; hasAlpha: boolean }> {
const { signal } = request;
if (signal?.aborted) throw abortError();
if (!Number.isFinite(request.fps) || request.fps < 1 || request.fps > MAX_FPS)
throw new MotionTemplateError("invalid_size", `fps must be 1..${MAX_FPS}`);
if (!path.isAbsolute(request.outPath) || path.extname(request.outPath).toLowerCase() !== ".mov")
throw new MotionTemplateError("invalid_asset", "outPath must be an absolute .mov path");
const outPath = await this.checkedPath(request.outPath);
const fontFiles: string[] = [];
for (const font of request.assets.fonts) {
const file = await this.checkedPath(font.path);
if (!FONT_EXTENSIONS.has(path.extname(file).toLowerCase()) || !(await stat(file)).isFile())
throw new MotionTemplateError("invalid_asset", "fonts must be TTF, OTF, or TTC files");
fontFiles.push(file);
}
const logos: string[] = [];
for (const logo of request.assets.logoPaths) {
logos.push(await loadLogoDataUri(await this.checkedPath(logo)));
}
const { template, ctx } = buildFrameContext({
template: request.template,
kit: request.brandKit,
width: request.width,
height: request.height,
logos,
});
const frames = frameCount(ctx.durationMs, request.fps);
if (ctx.width * ctx.height * frames > this.maxPixelFrames)
throw new MotionTemplateError("invalid_size", "motion clip exceeds the pixel budget");
const raster: RasterizeOptions = {
fontFiles,
defaultFontFamily: ctx.fonts.body,
loadSystemFonts: true,
};
const partial = path.join(
path.dirname(outPath),
`.${path.basename(outPath, ".mov")}.${randomUUID()}.partial.mov`,
);
const child = spawn(
this.ffmpegPath,
[
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-f",
"image2pipe",
"-framerate",
String(request.fps),
"-c:v",
"png",
"-i",
"pipe:0",
"-an",
"-c:v",
"qtrle",
"-pix_fmt",
"argb",
"-f",
"mov",
partial,
],
{ stdio: ["pipe", "ignore", "pipe"], shell: false },
);
const exited = waitForExit(child);
const kill = () => child.kill("SIGKILL");
signal?.addEventListener("abort", kill, { once: true });
try {
const stdin = child.stdin!;
stdin.on("error", () => {});
for (let index = 0; index < frames; index += 1) {
if (signal?.aborted) throw abortError();
const svg = template.frame((index * 1000) / request.fps, ctx);
const png = (await rasterizeFrame(svg, raster)).asPng();
if (signal?.aborted) throw abortError();
if (!stdin.write(png)) {
const result = await Promise.race([once(stdin, "drain").then(() => null), exited]);
if (result) throw ffmpegError(result);
}
}
stdin.end();
const result = await exited;
if (signal?.aborted) throw abortError();
if (result.code !== 0) throw ffmpegError(result);
await rename(partial, outPath);
return { path: outPath, hasAlpha: true };
} catch (error) {
kill();
await exited;
await rm(partial, { force: true });
throw signal?.aborted ? abortError() : error;
} finally {
signal?.removeEventListener("abort", kill);
}
}
private async checkedPath(file: string): Promise<string> {
if (!path.isAbsolute(file) || file.includes("\0"))
throw new MotionTemplateError("invalid_asset", "asset paths must be absolute");
return this.options.root ? assertInside(this.options.root, file) : path.resolve(file);
}
}
interface ExitResult {
code: number | null;
stderr: string;
}
function waitForExit(child: ChildProcess): Promise<ExitResult> {
let stderr = "";
child.stderr?.setEncoding("utf8");
child.stderr?.on("data", (chunk: string) => {
stderr = (stderr + chunk).slice(-4000);
});
return new Promise((resolve) => {
child.once("error", (error) => resolve({ code: -1, stderr: error.message }));
child.once("close", (code) => resolve({ code, stderr }));
});
}
function ffmpegError(result: ExitResult): Error {
return new Error(`ffmpeg failed (${result.code}): ${result.stderr.trim()}`);
}

View File

@ -0,0 +1,441 @@
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type { BrandKit, MotionTemplateRef } from "@rakazo/contracts";
import { MotionTemplateInfoSchema } from "@rakazo/contracts";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import type { MotionTemplate } from "./index.js";
import {
buildFrameContext,
loadLogoDataUri,
MOTION_TEMPLATES,
MotionTemplateError,
motionTemplate,
ResvgMotionGraphicsRenderer,
rasterizeFrame,
} from "./index.js";
import { GLITCH_BURST_INTERVAL_MS } from "./templates/intros.js";
const DEJAVU_DIR = "/usr/share/fonts/truetype/dejavu";
const DEJAVU = ["DejaVuSans.ttf", "DejaVuSans-Bold.ttf", "DejaVuSerif.ttf"].map((name) =>
path.join(DEJAVU_DIR, name),
);
const hasDejaVu = DEJAVU.every((file) => existsSync(file));
const hasFfmpeg = (() => {
try {
execFileSync("ffmpeg", ["-hide_banner", "-encoders"], { stdio: "pipe" });
execFileSync("ffprobe", ["-version"], { stdio: "pipe" });
return true;
} catch {
return false;
}
})();
const LOGO_SVG =
'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">' +
'<rect x="10" y="10" width="80" height="80" rx="20" fill="#ff8800"/>' +
'<circle cx="50" cy="50" r="22" fill="#ffffff"/></svg>';
const kit: BrandKit = {
name: "Test Brand",
colors: {
primary: "#2255cc",
secondary: "#9999aa",
accent: "#ff8800",
text: "#ffffff",
background: "#101820",
},
fonts: [{ family: "DejaVu Sans", artifactId: "font-1", role: "heading" }],
logoArtifactIds: ["logo-1"],
captionPresets: [],
musicBeds: [],
};
/** Parameters that exercise every template with all its text slots filled. */
const SAMPLE_PARAMS: Record<string, MotionTemplateRef["params"]> = {
intro_logo_reveal: { tagline: "Make it simple" },
intro_kinetic_type: { title: "Bold ideas move fast", tagline: "Since 2026" },
intro_glitch: { tagline: "Energy" },
intro_elegant_fade: { tagline: "Crafted slowly" },
outro_logo: { message: "Thanks" },
outro_subscribe: { cta: "Subscribe", handle: "@example" },
lt_clean: { title: "Alex Example", subtitle: "Host" },
lt_bar: { title: "Alex Example", subtitle: "Host", barColor: "accent" },
lt_pill: { title: "Alex Example", subtitle: "Host" },
caption_clean: { text: "A caption in the clean style" },
caption_karaoke: { text: "A caption in the karaoke style" },
caption_bold_social: { text: "Bold social caption" },
endcard_cta: { headline: "Get the full video", cta: "Watch now", price: "Only 9 today" },
endcard_minimal: { url: "example.com" },
};
let dir: string;
let logoPath: string;
let logo: string;
beforeAll(async () => {
dir = await mkdtemp(path.join(tmpdir(), "rakazo-motion-"));
logoPath = path.join(dir, "logo.svg");
await writeFile(logoPath, LOGO_SVG);
logo = await loadLogoDataUri(logoPath);
});
afterAll(async () => {
await rm(dir, { recursive: true, force: true });
});
function context(
template: MotionTemplate,
overrides: {
params?: MotionTemplateRef["params"];
durationMs?: number;
size?: [number, number];
} = {},
) {
const [width, height] = overrides.size ?? [640, 360];
return buildFrameContext({
template: {
templateId: template.info.id,
params: overrides.params ?? SAMPLE_PARAMS[template.info.id] ?? {},
durationMs: overrides.durationMs ?? template.info.minDurationMs,
},
kit,
width,
height,
logos: [logo],
}).ctx;
}
const raster = { fontFiles: DEJAVU, defaultFontFamily: "DejaVu Sans", loadSystemFonts: false };
/** Mean relative luminance of an RGBA frame composited over a solid backdrop. */
function meanLuminance(pixels: Buffer, backdrop: 0 | 1): number {
let sum = 0;
const count = pixels.length / 4;
for (let index = 0; index < pixels.length; index += 4) {
const alpha = pixels[index + 3]! / 255;
const channel = (offset: number) => {
const value = (pixels[index + offset]! / 255) * alpha + backdrop * (1 - alpha);
return value <= 0.03928 ? value / 12.92 : ((value + 0.055) / 1.055) ** 2.4;
};
sum += 0.2126 * channel(0) + 0.7152 * channel(1) + 0.0722 * channel(2);
}
return sum / count;
}
/**
* Count brightness reversals of at least 10% relative luminance (the WCAG 2.3.1
* flash threshold) per one-second window and return the worst window. A flash
* is a pair of opposing changes, so a steady fade counts as none; counting each
* reversal on its own is stricter than WCAG's three flashes per second.
*/
function maxReversalsPerSecond(series: number[], fps: number): number {
const transitions: number[] = [];
let direction = 0;
let extreme = series[0] ?? 0;
series.forEach((value, index) => {
const delta = value - extreme;
if (direction !== 0 && Math.sign(delta) === direction) {
extreme = value;
} else if (Math.abs(delta) >= 0.1) {
if (direction !== 0) transitions.push(index / fps);
direction = Math.sign(delta);
extreme = value;
}
});
let worst = 0;
for (const start of transitions) {
worst = Math.max(worst, transitions.filter((at) => at >= start && at < start + 1).length);
}
return worst;
}
describe("motion templates", () => {
it("ships the required set with valid, unique descriptors", () => {
const ids = MOTION_TEMPLATES.map((template) => template.info.id);
expect(new Set(ids).size).toBe(ids.length);
for (const template of MOTION_TEMPLATES) MotionTemplateInfoSchema.parse(template.info);
const count = (kind: string) =>
MOTION_TEMPLATES.filter((template) => template.info.kind === kind).length;
expect([
count("intro"),
count("outro"),
count("lower_third"),
count("caption"),
count("end_card"),
]).toEqual([4, 2, 3, 3, 2]);
});
it.each(MOTION_TEMPLATES.map((template) => [template.info.id, template] as const))(
"%s renders its first, middle and last frame",
async (_id, template) => {
const ctx = context(template);
for (const t of [0, ctx.durationMs / 2, ctx.durationMs]) {
const svg = template.frame(t, ctx);
expect(svg.startsWith("<svg")).toBe(true);
expect(svg).not.toMatch(/NaN|undefined|Infinity/);
const image = await rasterizeFrame(svg, { fontFiles: [], loadSystemFonts: true });
expect([image.width, image.height]).toEqual([640, 360]);
}
},
);
it("takes every color from the brand kit", () => {
const allowed = new Set(Object.values(kit.colors).map((color) => color.toLowerCase()));
for (const template of MOTION_TEMPLATES) {
const ctx = context(template);
const svg = template.frame(ctx.durationMs / 2, ctx);
const colors = [...svg.matchAll(/(?:fill|stroke)="([^"]+)"/g)].map((match) => match[1]!);
for (const color of colors) {
if (color === "none" || color.startsWith("url(")) continue;
expect(allowed.has(color.toLowerCase()), `${template.info.id} uses ${color}`).toBe(true);
}
}
});
it.skipIf(!hasDejaVu)("renders deterministic middle frames", async () => {
const hashes: Record<string, string> = {};
for (const template of MOTION_TEMPLATES) {
const ctx = context(template);
const image = await rasterizeFrame(template.frame(ctx.durationMs / 2, ctx), raster);
hashes[template.info.id] = createHash("sha256")
.update(image.pixels)
.digest("hex")
.slice(0, 16);
}
expect(hashes).toMatchInlineSnapshot(`
{
"caption_bold_social": "4a132766b59cdf12",
"caption_clean": "6194d5224d8e97ec",
"caption_karaoke": "f2db9bc313e4ccea",
"endcard_cta": "943fa754c9a30803",
"endcard_minimal": "be2b9f833895bc36",
"intro_elegant_fade": "8f8a98930994243f",
"intro_glitch": "35074c7ad30d26bb",
"intro_kinetic_type": "52129b6c4f5581e3",
"intro_logo_reveal": "2baf101db044941b",
"lt_bar": "a13184996d2ffc2e",
"lt_clean": "ca6a4239283e111e",
"lt_pill": "30a7342e49735d69",
"outro_logo": "84b1cd23213f8a2e",
"outro_subscribe": "18733abc1b0f1296",
}
`);
});
it("never changes full-frame brightness more than three times a second in the glitch intro", async () => {
const template = motionTemplate("intro_glitch");
const ctx = context(template, { durationMs: 4000, size: [160, 90] });
const fps = 25;
const dark: number[] = [];
const light: number[] = [];
for (let index = 0; index <= (ctx.durationMs * fps) / 1000; index += 1) {
const image = await rasterizeFrame(template.frame((index * 1000) / fps, ctx), {
fontFiles: [],
loadSystemFonts: true,
});
dark.push(meanLuminance(image.pixels, 0));
light.push(meanLuminance(image.pixels, 1));
}
expect(1000 / GLITCH_BURST_INTERVAL_MS).toBeLessThanOrEqual(3);
expect(maxReversalsPerSecond(dark, fps)).toBeLessThanOrEqual(3);
expect(maxReversalsPerSecond(light, fps)).toBeLessThanOrEqual(3);
});
it("detects a frame that does flash too fast", () => {
const flashing = Array.from({ length: 50 }, (_, index) => (index % 5 < 2 ? 0.9 : 0.05));
expect(maxReversalsPerSecond(flashing, 50)).toBeGreaterThan(3);
});
});
describe("motion params and escaping", () => {
const lowerThird = motionTemplate("lt_clean");
it.each([
[{ title: "x", extra: "y" }, "unknown_param"],
[{ title: 3 }, "invalid_param"],
[{ title: "a\u0007b" }, "invalid_param"],
[{ subtitle: "only" }, "missing_param"],
] as const)("rejects %j", (params, code) => {
expect(() => context(lowerThird, { params })).toThrow(
expect.objectContaining({ code }) as unknown as Error,
);
});
it("rejects oversized strings, unknown colors, durations, and frame sizes", () => {
expect(() => context(lowerThird, { params: { title: "x".repeat(201) } })).toThrow();
expect(() =>
context(motionTemplate("lt_bar"), { params: { title: "x", barColor: "#ff0000" } }),
).toThrow(MotionTemplateError);
expect(() => context(lowerThird, { durationMs: 60_000 })).toThrow(MotionTemplateError);
expect(() => context(lowerThird, { size: [8000, 360] })).toThrow(MotionTemplateError);
expect(() =>
buildFrameContext({
template: { templateId: "nope", params: {}, durationMs: 2000 },
kit,
width: 640,
height: 360,
logos: [],
}),
).toThrow(MotionTemplateError);
});
it("escapes text and keeps hostile input out of the markup", async () => {
const hostile = `<script>alert("x")</script> & 'quotes' ]]> <!--`;
for (const template of MOTION_TEMPLATES) {
const params = Object.fromEntries(
template.info.params
.filter((param) => param.type === "string")
.map((param) => [param.name, hostile]),
);
const ctx = context(template, { params });
const svg = template.frame(ctx.durationMs / 2, ctx);
expect(svg).not.toContain("<script");
expect(svg).not.toContain("<!--");
await rasterizeFrame(svg, { fontFiles: [], loadSystemFonts: true });
}
const ctx = context(lowerThird, { params: { title: hostile } });
expect(lowerThird.frame(1000, ctx)).toContain("&lt;script&gt;alert(&quot;x&quot;)");
});
it("shortens very long text instead of overflowing", () => {
const long = "W".repeat(200);
const ctx = context(lowerThird, { params: { title: long } });
const svg = lowerThird.frame(1000, ctx);
expect(svg).toContain("…");
expect(svg).not.toContain(long);
});
it("strips CSS-breaking characters from font families", () => {
const ctx = buildFrameContext({
template: { templateId: "lt_clean", params: { title: "x" }, durationMs: 2000 },
kit: {
...kit,
fonts: [{ family: `Evil", url(x); "`, artifactId: "font-1", role: "heading" }],
},
width: 640,
height: 360,
logos: [],
}).ctx;
expect(ctx.fonts.heading).toBe("Evil urlx");
});
it("accepts only self-contained PNG and SVG logos", async () => {
const jpg = path.join(dir, "logo.jpg");
const fakePng = path.join(dir, "fake.png");
const linked = path.join(dir, "linked.svg");
await writeFile(jpg, "x");
await writeFile(fakePng, "not a png");
await writeFile(
linked,
'<svg xmlns="http://www.w3.org/2000/svg"><image href="file:///etc/passwd"/></svg>',
);
for (const file of [jpg, fakePng, linked]) {
await expect(loadLogoDataUri(file)).rejects.toBeInstanceOf(MotionTemplateError);
}
expect(logo.startsWith("data:image/svg+xml;base64,")).toBe(true);
});
});
describe("ResvgMotionGraphicsRenderer", () => {
const request = (
outPath: string,
overrides: Partial<Parameters<ResvgMotionGraphicsRenderer["render"]>[0]> = {},
) => ({
template: { templateId: "intro_logo_reveal", params: { tagline: "Hello" }, durationMs: 1500 },
brandKit: kit,
assets: {
fonts: hasDejaVu ? [{ family: "DejaVu Sans", path: DEJAVU[0]! }] : [],
logoPaths: [logoPath],
},
width: 640,
height: 360,
fps: 25,
outPath,
...overrides,
});
it("lists its templates", () => {
expect(new ResvgMotionGraphicsRenderer().templates()).toHaveLength(MOTION_TEMPLATES.length);
});
it("refuses clips over the pixel budget before starting ffmpeg", async () => {
const renderer = new ResvgMotionGraphicsRenderer({
maxPixelFrames: 640 * 360 * 10,
ffmpegPath: "/nonexistent",
});
await expect(renderer.render(request(path.join(dir, "big.mov")))).rejects.toMatchObject({
code: "invalid_size",
});
});
it("refuses paths outside its root and non-MOV outputs", async () => {
const root = await mkdtemp(path.join(tmpdir(), "rakazo-motion-root-"));
try {
const renderer = new ResvgMotionGraphicsRenderer({ root, ffmpegPath: "/nonexistent" });
await expect(renderer.render(request(path.join(root, "out.mov")))).rejects.toThrow(/outside/);
await expect(renderer.render(request(path.join(root, "out.mp4")))).rejects.toMatchObject({
code: "invalid_asset",
});
} finally {
await rm(root, { recursive: true, force: true });
}
});
it.skipIf(!hasFfmpeg)("renders a MOV with alpha of the requested length", async () => {
const outPath = path.join(dir, "intro.mov");
const result = await new ResvgMotionGraphicsRenderer().render(request(outPath));
expect(result).toEqual({ path: outPath, hasAlpha: true });
const probe = JSON.parse(
execFileSync("ffprobe", [
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=codec_name,pix_fmt,width,height:format=duration",
"-of",
"json",
outPath,
]).toString(),
) as {
streams: Array<{ codec_name: string; pix_fmt: string; width: number; height: number }>;
format: { duration: string };
};
const stream = probe.streams[0]!;
expect(["qtrle", "prores"]).toContain(stream.codec_name);
expect(stream.pix_fmt).toMatch(/^(argb|rgba|bgra|yuva)/);
expect([stream.width, stream.height]).toEqual([640, 360]);
expect(Math.abs(Number(probe.format.duration) - 1.5)).toBeLessThanOrEqual(1 / 25);
expect((await readdir(dir)).filter((name) => name.includes("partial"))).toEqual([]);
});
it.skipIf(!hasFfmpeg)("cleans up when aborted mid-render", async () => {
const outDir = await mkdtemp(path.join(tmpdir(), "rakazo-motion-abort-"));
try {
const controller = new AbortController();
const outPath = path.join(outDir, "intro.mov");
const render = new ResvgMotionGraphicsRenderer().render(
request(outPath, {
template: { templateId: "intro_glitch", params: {}, durationMs: 4000 },
width: 1920,
height: 1080,
fps: 30,
signal: controller.signal,
}),
);
// Abort once ffmpeg has started writing its partial output.
for (let tries = 0; tries < 200 && (await readdir(outDir)).length === 0; tries += 1) {
await new Promise((resolve) => setTimeout(resolve, 10));
}
controller.abort();
await expect(render).rejects.toMatchObject({ name: "AbortError" });
expect(await readdir(outDir)).toEqual([]);
} finally {
await rm(outDir, { recursive: true, force: true });
}
});
});

View File

@ -0,0 +1,305 @@
import type { BrandKit, MotionTemplateInfo, MotionTemplateRef } from "@rakazo/contracts";
/*
* Shared building blocks for motion templates. A template is a pure function
* from (time, context) to a complete SVG document; everything visual comes from
* the brand kit in the context, and every piece of text passes through
* escapeXml before it reaches markup.
*/
export type BrandColorRole = keyof BrandKit["colors"];
export const BRAND_COLOR_ROLES: readonly BrandColorRole[] = [
"primary",
"secondary",
"accent",
"text",
"background",
];
export type MotionParamValue = string | number | boolean;
export interface MotionFrameContext {
width: number;
height: number;
durationMs: number;
kit: BrandKit;
/** Validated against the template's parameter list. */
params: Readonly<Record<string, MotionParamValue>>;
/** Logo data URIs in brand-kit order. */
logos: readonly string[];
/** Sanitized font families by role, already falling back across roles. */
fonts: { heading: string | null; body: string | null; caption: string | null };
}
export interface MotionTemplate {
info: MotionTemplateInfo;
frame(t: number, ctx: MotionFrameContext): string;
}
export class MotionTemplateError extends Error {
constructor(
readonly code:
| "unknown_template"
| "unknown_param"
| "invalid_param"
| "missing_param"
| "invalid_duration"
| "invalid_size"
| "invalid_asset",
message: string,
) {
super(message);
this.name = "MotionTemplateError";
}
}
export const MAX_PARAM_STRING_LENGTH = 200;
const MAX_PARAM_NUMBER = 1_000_000;
// XML 1.0 cannot carry these at all, escaped or not.
// biome-ignore lint/suspicious/noControlCharactersInRegex: the point is to find them.
const XML_FORBIDDEN = /[\u0000-\u0008\u000B\u000C\u000E-\u001F￾￿]/;
const LONE_SURROGATE = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;
/** Check a template reference's params against its declared list. */
export function validateParams(
info: MotionTemplateInfo,
params: MotionTemplateRef["params"],
): Record<string, MotionParamValue> {
const declared = new Map(info.params.map((param) => [param.name, param]));
const result: Record<string, MotionParamValue> = {};
for (const [name, value] of Object.entries(params)) {
const spec = declared.get(name);
if (!spec) throw new MotionTemplateError("unknown_param", `${info.id} has no param "${name}"`);
if (spec.type === "string" || spec.type === "color") {
if (typeof value !== "string")
throw new MotionTemplateError("invalid_param", `${name} must be a string`);
if (value.length > MAX_PARAM_STRING_LENGTH)
throw new MotionTemplateError("invalid_param", `${name} is too long`);
if (XML_FORBIDDEN.test(value) || LONE_SURROGATE.test(value))
throw new MotionTemplateError("invalid_param", `${name} contains unsupported characters`);
if (spec.type === "color" && !BRAND_COLOR_ROLES.includes(value as BrandColorRole))
throw new MotionTemplateError(
"invalid_param",
`${name} must name a brand color: ${BRAND_COLOR_ROLES.join(", ")}`,
);
const trimmed = value.trim();
if (trimmed === "") continue;
result[name] = trimmed;
} else if (spec.type === "number") {
if (
typeof value !== "number" ||
!Number.isFinite(value) ||
Math.abs(value) > MAX_PARAM_NUMBER
)
throw new MotionTemplateError("invalid_param", `${name} must be a finite number`);
result[name] = value;
} else {
if (typeof value !== "boolean")
throw new MotionTemplateError("invalid_param", `${name} must be a boolean`);
result[name] = value;
}
}
for (const spec of info.params) {
if (spec.required && !(spec.name in result))
throw new MotionTemplateError("missing_param", `${info.id} needs "${spec.name}"`);
}
return result;
}
export function validateDuration(info: MotionTemplateInfo, durationMs: number): void {
if (
!Number.isInteger(durationMs) ||
durationMs < info.minDurationMs ||
durationMs > info.maxDurationMs
)
throw new MotionTemplateError(
"invalid_duration",
`${info.id} runs ${info.minDurationMs}..${info.maxDurationMs} ms`,
);
}
/** Font family names reach a CSS list inside an attribute; keep them plain. */
export function sanitizeFontFamily(family: string): string | null {
const clean = family
.replace(/[^\p{L}\p{N} _-]/gu, "")
.replace(/\s+/g, " ")
.trim();
return clean === "" ? null : clean.slice(0, 80);
}
export function brandFonts(kit: BrandKit): MotionFrameContext["fonts"] {
const byRole = (role: "heading" | "body" | "caption") => {
const font = kit.fonts.find((entry) => entry.role === role);
return font ? sanitizeFontFamily(font.family) : null;
};
const heading = byRole("heading");
const body = byRole("body");
const caption = byRole("caption");
return {
heading: heading ?? body ?? caption,
body: body ?? heading ?? caption,
caption: caption ?? body ?? heading,
};
}
export function escapeXml(value: string): string {
return value
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;")
.replace(/'/g, "&apos;");
}
/** Round for stable, compact markup. */
export function num(value: number): string {
const rounded = Math.round(value * 100) / 100;
return Object.is(rounded, -0) ? "0" : String(rounded);
}
type Attrs = Record<string, string | number | undefined>;
function attrs(values: Attrs): string {
return Object.entries(values)
.filter((entry): entry is [string, string | number] => entry[1] !== undefined)
.map(([key, value]) => ` ${key}="${typeof value === "number" ? num(value) : escapeXml(value)}"`)
.join("");
}
export function el(name: string, values: Attrs, children = ""): string {
return children === ""
? `<${name}${attrs(values)}/>`
: `<${name}${attrs(values)}>${children}</${name}>`;
}
export function fontStack(family: string | null, generic: "sans-serif" | "serif"): string {
return family ? `${family}, ${generic}` : generic;
}
/** Rough advance width per character relative to font size; deliberately generous. */
const CHAR_WIDTH = 0.6;
/**
* Pick a font size so `text` fits `maxWidth` on one line, shrinking down to
* `minSize`; anything still too wide is cut with an ellipsis.
*/
export function fitText(
text: string,
maxWidth: number,
size: number,
minSize: number,
letterSpacingEm = 0,
): { text: string; size: number } {
const chars = [...text];
const per = CHAR_WIDTH + letterSpacingEm;
const fitted = maxWidth / Math.max(1, chars.length * per);
if (fitted >= size) return { text, size };
if (fitted >= minSize) return { text, size: fitted };
const keep = Math.max(1, Math.floor(maxWidth / (minSize * per)) - 1);
return { text: `${chars.slice(0, keep).join("").trimEnd()}…`, size: minSize };
}
export function textEl(content: string, values: Attrs): string {
return el("text", values, escapeXml(content) || " ");
}
export function stringParam(ctx: MotionFrameContext, name: string): string | null {
const value = ctx.params[name];
return typeof value === "string" ? value : null;
}
export function colorParam(
ctx: MotionFrameContext,
name: string,
fallback: BrandColorRole,
): string {
const role = stringParam(ctx, name);
return ctx.kit.colors[
role && BRAND_COLOR_ROLES.includes(role as BrandColorRole) ? (role as BrandColorRole) : fallback
];
}
export function logoEl(
ctx: MotionFrameContext,
box: { x: number; y: number; w: number; h: number },
extra: Attrs = {},
): string {
const href = ctx.logos[0];
if (!href) return "";
return el("image", {
href,
x: box.x,
y: box.y,
width: box.w,
height: box.h,
preserveAspectRatio: "xMidYMid meet",
...extra,
});
}
export function svgDocument(ctx: MotionFrameContext, body: string, defs = ""): string {
const { width, height } = ctx;
return (
`<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"` +
` width="${width}" height="${height}" viewBox="0 0 ${width} ${height}">` +
(defs ? `<defs>${defs}</defs>` : "") +
body +
"</svg>"
);
}
export function fullBackground(ctx: MotionFrameContext, opacity = 1): string {
return el("rect", {
x: 0,
y: 0,
width: ctx.width,
height: ctx.height,
fill: ctx.kit.colors.background,
opacity: opacity < 1 ? opacity : undefined,
});
}
/** A filled, centered call-to-action button in the primary brand color. */
export function ctaButton(
ctx: MotionFrameContext,
label: string,
cx: number,
cy: number,
scale = 1,
opacity = 1,
): string {
const u = unit(ctx);
const fitted = fitText(label, ctx.width * 0.7, u * 0.07, u * 0.035);
const width = Math.min(ctx.width * 0.8, [...fitted.text].length * fitted.size * 0.6 + u * 0.16);
const height = fitted.size * 1.9;
return el(
"g",
{
transform: `translate(${num(cx)} ${num(cy)}) scale(${num(Math.max(scale, 0.001))})`,
opacity: opacity < 1 ? opacity : undefined,
},
el("rect", {
x: -width / 2,
y: -height / 2,
width,
height,
rx: height / 2,
fill: ctx.kit.colors.primary,
}) +
textEl(fitted.text, {
x: 0,
y: 0,
"text-anchor": "middle",
"dominant-baseline": "central",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-weight": 700,
"font-size": fitted.size,
fill: ctx.kit.colors.background,
}),
);
}
/** Short side of the frame; templates scale type and spacing from it. */
export function unit(ctx: MotionFrameContext): number {
return Math.min(ctx.width, ctx.height);
}

View File

@ -0,0 +1,182 @@
import type { CaptionStylePreset } from "@rakazo/contracts";
import { easeInCubic, easeOutBack, easeOutCubic, progress } from "../easing.js";
import type { MotionFrameContext, MotionTemplate } from "../template.js";
import {
el,
escapeXml,
fontStack,
num,
sanitizeFontFamily,
stringParam,
svgDocument,
} from "../template.js";
/*
* Preview cards for caption presets (brand-kit editor, caption picker). The
* burned-in captions of a render come from the compiler's ASS subtitles; these
* only need to look like them.
*/
const CAPTION = { kind: "caption", minDurationMs: 1000, maxDurationMs: 10000 } as const;
const CAPTION_PARAMS = [
{ name: "text", type: "string", required: true },
{ name: "presetId", type: "string", required: false },
] as const;
const MAX_LINES = 2;
interface CaptionStyle {
family: string | null;
size: number;
text: string;
highlight: string;
background: string | null;
uppercase: boolean;
}
function captionStyle(
ctx: MotionFrameContext,
template: CaptionStylePreset["template"],
): CaptionStyle {
const presetId = stringParam(ctx, "presetId");
const presets = ctx.kit.captionPresets;
const preset =
(presetId ? presets.find((entry) => entry.id === presetId) : undefined) ??
presets.find((entry) => entry.template === template);
const { colors } = ctx.kit;
return {
family: preset ? sanitizeFontFamily(preset.fontFamily) : ctx.fonts.caption,
size: (ctx.height * (preset?.sizePct ?? (template === "bold_social" ? 7 : 5.5))) / 100,
text: preset?.textColor ?? colors.text,
highlight: preset?.highlightColor ?? colors.accent,
background: preset
? (preset.backgroundColor ?? null)
: template === "clean"
? colors.background
: null,
uppercase: preset?.uppercase ?? template === "bold_social",
};
}
/** Greedy word wrap into at most MAX_LINES lines that fit `maxWidth`. */
function wrapWords(words: string[], size: number, maxWidth: number): string[][] {
const maxChars = Math.max(4, Math.floor(maxWidth / (size * 0.6)));
const lines: string[][] = [[]];
let length = 0;
for (const raw of words) {
const word = [...raw].length > maxChars ? `${[...raw].slice(0, maxChars - 1).join("")}…` : raw;
const current = lines[lines.length - 1]!;
const added = (current.length > 0 ? 1 : 0) + [...word].length;
if (current.length > 0 && length + added > maxChars) {
if (lines.length === MAX_LINES) {
current[current.length - 1] = `${current[current.length - 1]}…`;
return lines;
}
lines.push([word]);
length = [...word].length;
} else {
current.push(word);
length += added;
}
}
return lines;
}
function captionFrame(
t: number,
ctx: MotionFrameContext,
template: CaptionStylePreset["template"],
): string {
const style = captionStyle(ctx, template);
const raw = stringParam(ctx, "text") ?? "";
const words = (style.uppercase ? raw.toUpperCase() : raw).split(/\s+/).filter(Boolean);
const lines = wrapWords(words, style.size, ctx.width * 0.84);
const enter = progress(t, 0, Math.min(250, ctx.durationMs * 0.2));
const leave = easeInCubic(
progress(t, ctx.durationMs - Math.min(200, ctx.durationMs * 0.15), ctx.durationMs),
);
const total = lines.reduce((sum, line) => sum + line.length, 0);
// Karaoke and bold styles move a highlight through the words across the hold.
const active =
template === "clean"
? -1
: Math.min(
total - 1,
Math.floor(progress(t, ctx.durationMs * 0.1, ctx.durationMs * 0.9) * total),
);
const lineHeight = style.size * 1.25;
const baseY = ctx.height * 0.84 - (lines.length - 1) * lineHeight;
let wordIndex = 0;
const rendered = lines.map((line, row) => {
const spans = line.map((word, column) => {
const index = wordIndex++;
const fill =
template === "karaoke"
? index <= active
? style.highlight
: style.text
: template === "bold_social" && index === active
? style.highlight
: style.text;
return el("tspan", { fill }, escapeXml(column < line.length - 1 ? `${word} ` : word));
});
return el(
"text",
{
x: ctx.width / 2,
y: baseY + row * lineHeight,
"text-anchor": "middle",
"font-family": fontStack(style.family, "sans-serif"),
"font-weight": template === "bold_social" ? 900 : 600,
"font-size": style.size,
stroke: template === "bold_social" ? ctx.kit.colors.background : undefined,
"stroke-width": template === "bold_social" ? style.size * 0.12 : undefined,
"stroke-linejoin": template === "bold_social" ? "round" : undefined,
"paint-order": template === "bold_social" ? "stroke" : undefined,
},
spans.join("") || " ",
);
});
const longest = Math.max(1, ...lines.map((line) => [...line.join(" ")].length));
const plate = style.background
? el("rect", {
x: ctx.width / 2 - (longest * style.size * 0.6) / 2 - style.size * 0.5,
y: baseY - style.size * 1.05,
width: longest * style.size * 0.6 + style.size,
height: lines.length * lineHeight + style.size * 0.45,
rx: style.size * 0.25,
fill: style.background,
opacity: 0.75,
})
: "";
const scale = template === "bold_social" ? (enter <= 0 ? 0.001 : easeOutBack(enter)) : 1;
const originY = ctx.height * 0.84;
return svgDocument(
ctx,
el(
"g",
{
opacity: (template === "bold_social" ? 1 : easeOutCubic(enter)) * (1 - leave),
transform:
scale === 1
? undefined
: `translate(${num(ctx.width / 2)} ${num(originY)}) scale(${num(scale)}) translate(${num(-ctx.width / 2)} ${num(-originY)})`,
},
plate + rendered.join(""),
),
);
}
export const captionClean: MotionTemplate = {
info: { id: "caption_clean", name: "Clean", ...CAPTION, params: [...CAPTION_PARAMS] },
frame: (t, ctx) => captionFrame(t, ctx, "clean"),
};
export const captionKaraoke: MotionTemplate = {
info: { id: "caption_karaoke", name: "Karaoke", ...CAPTION, params: [...CAPTION_PARAMS] },
frame: (t, ctx) => captionFrame(t, ctx, "karaoke"),
};
export const captionBoldSocial: MotionTemplate = {
info: { id: "caption_bold_social", name: "Bold social", ...CAPTION, params: [...CAPTION_PARAMS] },
frame: (t, ctx) => captionFrame(t, ctx, "bold_social"),
};

View File

@ -0,0 +1,153 @@
import { easeOutBack, easeOutCubic, progress } from "../easing.js";
import type { MotionFrameContext, MotionTemplate } from "../template.js";
import {
ctaButton,
fitText,
fontStack,
fullBackground,
logoEl,
stringParam,
svgDocument,
textEl,
unit,
} from "../template.js";
const END_CARD = { kind: "end_card", minDurationMs: 3000, maxDurationMs: 10000 } as const;
/** Staggered rise-in for the n-th element of a card. */
function rise(t: number, ctx: MotionFrameContext, index: number): number {
const start = ctx.durationMs * (0.05 + index * 0.08);
return easeOutCubic(progress(t, start, start + ctx.durationMs * 0.18));
}
export const endCardCta: MotionTemplate = {
info: {
id: "endcard_cta",
name: "Call to action",
...END_CARD,
params: [
{ name: "headline", type: "string", required: true },
{ name: "cta", type: "string", required: true },
{ name: "price", type: "string", required: false },
],
},
frame(t, ctx) {
const u = unit(ctx);
const cx = ctx.width / 2;
const hasLogo = ctx.logos.length > 0;
const logoSize = u * 0.16;
const top = ctx.height * (hasLogo ? 0.2 : 0.3);
const logoAppear = rise(t, ctx, 0);
const logo = logoEl(
ctx,
{ x: cx - logoSize / 2, y: top, w: logoSize, h: logoSize },
{ opacity: logoAppear },
);
const headlineY = top + (hasLogo ? logoSize + u * 0.12 : u * 0.05);
const headline = fitText(
stringParam(ctx, "headline") ?? "",
ctx.width * 0.84,
u * 0.09,
u * 0.04,
);
const headlineAppear = rise(t, ctx, 1);
let y = headlineY;
let body =
fullBackground(ctx) +
logo +
textEl(headline.text, {
x: cx,
y: y + (1 - headlineAppear) * u * 0.04,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 800,
"font-size": headline.size,
fill: ctx.kit.colors.text,
opacity: headlineAppear,
});
const price = stringParam(ctx, "price");
if (price) {
const fitted = fitText(price, ctx.width * 0.7, u * 0.055, u * 0.03);
const appear = rise(t, ctx, 2);
y += u * 0.1;
body += textEl(fitted.text, {
x: cx,
y: y + (1 - appear) * u * 0.03,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-weight": 600,
"font-size": fitted.size,
fill: ctx.kit.colors.accent,
opacity: appear,
});
}
const pop = progress(t, ctx.durationMs * 0.3, ctx.durationMs * 0.45);
body += ctaButton(
ctx,
stringParam(ctx, "cta") ?? "",
cx,
y + u * 0.17,
pop <= 0 ? 0 : easeOutBack(pop),
);
return svgDocument(ctx, body);
},
};
export const endCardMinimal: MotionTemplate = {
info: {
id: "endcard_minimal",
name: "Minimal",
...END_CARD,
params: [
{ name: "headline", type: "string", required: false },
{ name: "url", type: "string", required: false },
],
},
frame(t, ctx) {
const u = unit(ctx);
const cx = ctx.width / 2;
const size = u * 0.22;
const cy = ctx.height * 0.4;
const appear = rise(t, ctx, 0);
const logo = logoEl(
ctx,
{ x: cx - size / 2, y: cy - size / 2, w: size, h: size },
{ opacity: appear },
);
const headline = fitText(
stringParam(ctx, "headline") ?? ctx.kit.name,
ctx.width * 0.8,
u * 0.075,
u * 0.035,
);
const headlineY = ctx.logos.length > 0 ? cy + size / 2 + u * 0.1 : cy + u * 0.04;
const headlineAppear = rise(t, ctx, 1);
let body =
fullBackground(ctx) +
logo +
textEl(headline.text, {
x: cx,
y: headlineY,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 600,
"font-size": headline.size,
fill: ctx.kit.colors.text,
opacity: headlineAppear,
});
const url = stringParam(ctx, "url");
if (url) {
const fitted = fitText(url, ctx.width * 0.8, u * 0.045, u * 0.025);
body += textEl(fitted.text, {
x: cx,
y: headlineY + u * 0.09,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": fitted.size,
fill: ctx.kit.colors.secondary,
opacity: rise(t, ctx, 2),
});
}
return svgDocument(ctx, body);
},
};

View File

@ -0,0 +1,309 @@
import {
clamp,
easeInOutCubic,
easeOutBack,
easeOutCubic,
easeOutExpo,
fadeEnvelope,
hash01,
lerp,
progress,
} from "../easing.js";
import type { MotionFrameContext, MotionTemplate } from "../template.js";
import {
el,
escapeXml,
fitText,
fontStack,
fullBackground,
logoEl,
num,
stringParam,
svgDocument,
textEl,
unit,
} from "../template.js";
const INTRO_PARAMS = [
{ name: "title", type: "string", required: false },
{ name: "tagline", type: "string", required: false },
] as const;
function title(ctx: MotionFrameContext): string {
return stringParam(ctx, "title") ?? ctx.kit.name;
}
/** Fades the whole intro, background included, so it hands over to the video. */
function outro(t: number, ctx: MotionFrameContext): number {
return fadeEnvelope(t, ctx.durationMs, 0, Math.min(400, ctx.durationMs * 0.15));
}
function tagline(ctx: MotionFrameContext, t: number, y: number, startPct: number): string {
const text = stringParam(ctx, "tagline");
if (!text) return "";
const appear = easeOutCubic(
progress(t, ctx.durationMs * startPct, ctx.durationMs * (startPct + 0.2)),
);
const fitted = fitText(text, ctx.width * 0.8, unit(ctx) * 0.055, unit(ctx) * 0.03);
return textEl(fitted.text, {
x: ctx.width / 2,
y: y + (1 - appear) * unit(ctx) * 0.03,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": fitted.size,
fill: ctx.kit.colors.text,
opacity: appear,
});
}
export const introLogoReveal: MotionTemplate = {
info: {
id: "intro_logo_reveal",
kind: "intro",
name: "Logo reveal",
minDurationMs: 1500,
maxDurationMs: 4000,
params: [...INTRO_PARAMS],
},
frame(t, ctx) {
const u = unit(ctx);
const cx = ctx.width / 2;
const cy = ctx.height * 0.46;
const d = ctx.durationMs;
const circle = easeOutExpo(progress(t, 0, d * 0.3));
const pop = progress(t, d * 0.18, d * 0.48);
const scale = pop <= 0 ? 0 : easeOutBack(pop);
const radius = u * 0.24;
let mark: string;
if (ctx.logos.length > 0) {
const size = radius * 1.3 * scale;
mark = logoEl(ctx, { x: cx - size / 2, y: cy - size / 2, w: size, h: size });
} else {
const fitted = fitText(title(ctx), ctx.width * 0.8, u * 0.14, u * 0.05);
mark = textEl(fitted.text, {
x: cx,
y: cy,
"text-anchor": "middle",
"dominant-baseline": "central",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 700,
"font-size": fitted.size * Math.max(scale, 0.001),
fill: ctx.kit.colors.text,
});
}
const ring = el("circle", {
cx,
cy,
r: Math.max(radius * circle, 0.01),
fill: ctx.logos.length > 0 ? ctx.kit.colors.primary : "none",
stroke: ctx.kit.colors.accent,
"stroke-width": u * 0.01,
opacity: ctx.logos.length > 0 ? 1 : 0.9,
});
const body = fullBackground(ctx) + ring + mark + tagline(ctx, t, cy + radius + u * 0.12, 0.42);
return svgDocument(ctx, el("g", { opacity: outro(t, ctx) }, body));
},
};
export const introKineticType: MotionTemplate = {
info: {
id: "intro_kinetic_type",
kind: "intro",
name: "Kinetic type",
minDurationMs: 1500,
maxDurationMs: 4000,
params: [...INTRO_PARAMS],
},
frame(t, ctx) {
const u = unit(ctx);
const d = ctx.durationMs;
const fitted = fitText(title(ctx), ctx.width * 0.86, u * 0.16, u * 0.05);
const words = fitted.text.split(/\s+/).filter(Boolean).slice(0, 12);
const stagger = (d * 0.35) / Math.max(words.length, 1);
const baseY = ctx.height * 0.48;
let previousOffset = 0;
const spans = words.map((word, index) => {
const appear = easeOutExpo(progress(t, index * stagger, index * stagger + d * 0.25));
const offset = (1 - appear) * u * 0.18;
const dy = offset - previousOffset;
previousOffset = offset;
const last = index === words.length - 1 && words.length > 1;
return el(
"tspan",
{
dy,
fill: last ? ctx.kit.colors.primary : ctx.kit.colors.text,
"fill-opacity": clamp(appear * 1.4),
},
escapeXml(index === words.length - 1 ? word : `${word} `),
);
});
const heading = el(
"text",
{
x: ctx.width / 2,
y: baseY,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 800,
"font-size": fitted.size,
},
spans.join("") || " ",
);
const barGrow = easeInOutCubic(progress(t, d * 0.3, d * 0.6));
const barWidth = ctx.width * 0.3 * barGrow;
const bar = el("rect", {
x: ctx.width / 2 - barWidth / 2,
y: baseY + fitted.size * 0.3,
width: barWidth,
height: u * 0.012,
fill: ctx.kit.colors.accent,
});
const body = fullBackground(ctx) + heading + bar + tagline(ctx, t, baseY + u * 0.16, 0.45);
return svgDocument(ctx, el("g", { opacity: outro(t, ctx) }, body));
},
};
/**
* Glitch bursts are scheduled at a fixed 2 Hz. Photosensitivity guidance
* (WCAG 2.3.1) allows at most three flashes per second; the template never
* changes full-frame brightness, and the bursts stay below that rate even if
* they were counted as flashes.
*/
export const GLITCH_BURST_INTERVAL_MS = 500;
const GLITCH_BURST_MS = 120;
const GLITCH_SLICES = 6;
export const introGlitch: MotionTemplate = {
info: {
id: "intro_glitch",
kind: "intro",
name: "Glitch",
minDurationMs: 1500,
maxDurationMs: 4000,
params: [...INTRO_PARAMS],
},
frame(t, ctx) {
const u = unit(ctx);
const d = ctx.durationMs;
const fitted = fitText(title(ctx).toUpperCase(), ctx.width * 0.84, u * 0.15, u * 0.05);
const cy = ctx.height * 0.47;
const burstIndex = Math.floor((t - 200) / GLITCH_BURST_INTERVAL_MS);
const inBurst =
t >= 200 && t - 200 - burstIndex * GLITCH_BURST_INTERVAL_MS < GLITCH_BURST_MS && t < d * 0.85;
const reveal = easeOutCubic(progress(t, 0, d * 0.25));
const textAttrs = {
x: ctx.width / 2,
y: cy,
"text-anchor": "middle",
"dominant-baseline": "central",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 900,
"font-size": fitted.size,
};
const bandTop = cy - fitted.size * 0.7;
const bandHeight = fitted.size * 1.4;
const sliceHeight = bandHeight / GLITCH_SLICES;
const defs = [
el(
"clipPath",
{ id: "reveal" },
el("rect", { x: 0, y: 0, width: ctx.width * reveal, height: ctx.height }),
),
...Array.from({ length: GLITCH_SLICES }, (_, index) =>
el(
"clipPath",
{ id: `slice${index}` },
el("rect", {
x: 0,
y: bandTop + index * sliceHeight,
width: ctx.width,
height: sliceHeight,
}),
),
),
].join("");
let title_: string;
if (inBurst) {
const split = u * 0.012;
const ghosts =
textEl(fitted.text, {
...textAttrs,
x: ctx.width / 2 - split,
fill: ctx.kit.colors.primary,
opacity: 0.8,
}) +
textEl(fitted.text, {
...textAttrs,
x: ctx.width / 2 + split,
fill: ctx.kit.colors.accent,
opacity: 0.8,
});
const slices = Array.from({ length: GLITCH_SLICES }, (_, index) => {
const shift = (hash01(burstIndex, index) - 0.5) * u * 0.08;
return el(
"g",
{ "clip-path": `url(#slice${index})`, transform: `translate(${num(shift)} 0)` },
textEl(fitted.text, { ...textAttrs, fill: ctx.kit.colors.text }),
);
}).join("");
title_ = ghosts + slices;
} else {
title_ = textEl(fitted.text, { ...textAttrs, fill: ctx.kit.colors.text });
}
const scan = el("rect", {
x: 0,
y: lerp(bandTop, bandTop + bandHeight, (t % 900) / 900),
width: ctx.width * reveal,
height: u * 0.004,
fill: ctx.kit.colors.accent,
opacity: 0.6,
});
const body =
fullBackground(ctx) +
el("g", { "clip-path": "url(#reveal)" }, title_ + scan) +
tagline(ctx, t, cy + fitted.size * 0.7 + u * 0.08, 0.4);
return svgDocument(ctx, el("g", { opacity: outro(t, ctx) }, body), defs);
},
};
export const introElegantFade: MotionTemplate = {
info: {
id: "intro_elegant_fade",
kind: "intro",
name: "Elegant fade",
minDurationMs: 1500,
maxDurationMs: 4000,
params: [...INTRO_PARAMS],
},
frame(t, ctx) {
const u = unit(ctx);
const d = ctx.durationMs;
const cy = ctx.height * 0.46;
const line = easeInOutCubic(progress(t, 0, d * 0.4));
const appear = easeOutCubic(progress(t, d * 0.15, d * 0.6));
const spacing = lerp(0.4, 0.15, appear);
const fitted = fitText(title(ctx), ctx.width * 0.8, u * 0.11, u * 0.04, spacing);
const heading = textEl(fitted.text, {
x: ctx.width / 2,
y: cy - u * 0.03 + (1 - appear) * u * 0.02,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.heading, "serif"),
"font-weight": 400,
"font-size": fitted.size,
"letter-spacing": `${num(spacing)}em`,
fill: ctx.kit.colors.text,
opacity: appear,
});
const lineWidth = ctx.width * 0.4 * line;
const rule = el("rect", {
x: ctx.width / 2 - lineWidth / 2,
y: cy + u * 0.02,
width: lineWidth,
height: Math.max(1, u * 0.003),
fill: ctx.kit.colors.accent,
});
const body = fullBackground(ctx) + heading + rule + tagline(ctx, t, cy + u * 0.12, 0.5);
return svgDocument(ctx, el("g", { opacity: outro(t, ctx) }, body));
},
};

View File

@ -0,0 +1,245 @@
import { easeInCubic, easeOutBack, easeOutCubic, progress } from "../easing.js";
import type { MotionFrameContext, MotionTemplate } from "../template.js";
import {
colorParam,
el,
fitText,
fontStack,
num,
stringParam,
svgDocument,
textEl,
unit,
} from "../template.js";
const LOWER_THIRD = {
kind: "lower_third",
minDurationMs: 2000,
maxDurationMs: 10000,
} as const;
const TEXT_PARAMS = [
{ name: "title", type: "string", required: true },
{ name: "subtitle", type: "string", required: false },
] as const;
/** 0 → 1 while entering, 1 while held, 1 → 0 while leaving. */
function presence(t: number, ctx: MotionFrameContext): { enter: number; leave: number } {
return {
enter: progress(t, 0, Math.min(600, ctx.durationMs * 0.25)),
leave: easeInCubic(
progress(t, ctx.durationMs - Math.min(400, ctx.durationMs * 0.2), ctx.durationMs),
),
};
}
function lines(ctx: MotionFrameContext, maxWidth: number) {
const u = unit(ctx);
const title = fitText(stringParam(ctx, "title") ?? "", maxWidth, u * 0.06, u * 0.03);
const subtitleText = stringParam(ctx, "subtitle");
const subtitle = subtitleText ? fitText(subtitleText, maxWidth, u * 0.04, u * 0.025) : null;
return { title, subtitle };
}
function textWidth(text: string, size: number): number {
return [...text].length * size * 0.6;
}
export const lowerThirdClean: MotionTemplate = {
info: { id: "lt_clean", name: "Clean", ...LOWER_THIRD, params: [...TEXT_PARAMS] },
frame(t, ctx) {
const u = unit(ctx);
const { enter, leave } = presence(t, ctx);
const x = ctx.width * 0.07;
const y = ctx.height * 0.8;
const { title, subtitle } = lines(ctx, ctx.width * 0.6);
const appear = easeOutCubic(enter) * (1 - leave);
const slide = (1 - easeOutCubic(enter)) * -u * 0.05;
const accentHeight = (title.size + (subtitle ? subtitle.size * 1.4 : 0)) * easeOutCubic(enter);
const plateWidth =
Math.max(
textWidth(title.text, title.size),
subtitle ? textWidth(subtitle.text, subtitle.size) : 0,
) +
u * 0.08;
const plateHeight = title.size * 1.5 + (subtitle ? subtitle.size * 1.5 : 0);
const body =
el("rect", {
x: x - u * 0.02,
y: y - title.size * 1.15,
width: plateWidth,
height: plateHeight,
rx: u * 0.008,
fill: ctx.kit.colors.background,
opacity: 0.55,
}) +
el("rect", {
x: x - u * 0.02,
y: y - title.size * 1.15,
width: u * 0.008,
height: accentHeight + title.size * 0.5,
fill: ctx.kit.colors.accent,
}) +
textEl(title.text, {
x: x + u * 0.01,
y,
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 700,
"font-size": title.size,
fill: ctx.kit.colors.text,
}) +
(subtitle
? textEl(subtitle.text, {
x: x + u * 0.01,
y: y + subtitle.size * 1.4,
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": subtitle.size,
fill: ctx.kit.colors.text,
})
: "");
return svgDocument(
ctx,
el("g", { opacity: appear, transform: `translate(${num(slide)} 0)` }, body),
);
},
};
export const lowerThirdBar: MotionTemplate = {
info: {
id: "lt_bar",
name: "Bar",
...LOWER_THIRD,
params: [...TEXT_PARAMS, { name: "barColor", type: "color", required: false }],
},
frame(t, ctx) {
const u = unit(ctx);
const { enter, leave } = presence(t, ctx);
const x = ctx.width * 0.06;
const y = ctx.height * 0.76;
const { title, subtitle } = lines(ctx, ctx.width * 0.6);
const sweep = easeOutCubic(enter) * (1 - leave);
const barWidth = (textWidth(title.text, title.size) + u * 0.08) * sweep;
const barHeight = title.size * 1.6;
const subWidth = subtitle ? (textWidth(subtitle.text, subtitle.size) + u * 0.06) * sweep : 0;
const textOpacity =
easeOutCubic(progress(t, ctx.durationMs * 0.08, ctx.durationMs * 0.2)) * (1 - leave);
const body =
el("clipPath", { id: "bar" }, el("rect", { x, y, width: barWidth, height: barHeight })) +
el("rect", {
x,
y,
width: barWidth,
height: barHeight,
fill: colorParam(ctx, "barColor", "primary"),
}) +
textEl(title.text, {
x: x + u * 0.04,
y: y + barHeight / 2,
"dominant-baseline": "central",
"clip-path": "url(#bar)",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 800,
"font-size": title.size,
fill: ctx.kit.colors.background,
opacity: textOpacity,
}) +
(subtitle
? el("rect", {
x,
y: y + barHeight,
width: subWidth,
height: subtitle.size * 1.6,
fill: ctx.kit.colors.background,
}) +
textEl(subtitle.text, {
x: x + u * 0.03,
y: y + barHeight + subtitle.size * 0.8,
"dominant-baseline": "central",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": subtitle.size,
fill: ctx.kit.colors.text,
opacity: textOpacity,
})
: "");
return svgDocument(ctx, body);
},
};
export const lowerThirdPill: MotionTemplate = {
info: { id: "lt_pill", name: "Pill", ...LOWER_THIRD, params: [...TEXT_PARAMS] },
frame(t, ctx) {
const u = unit(ctx);
const { enter, leave } = presence(t, ctx);
const { title, subtitle } = lines(ctx, ctx.width * 0.55);
const height = subtitle ? title.size * 1.3 + subtitle.size * 1.6 : title.size * 2;
const dot = height * 0.62;
const width =
dot +
u * 0.07 +
Math.max(
textWidth(title.text, title.size),
subtitle ? textWidth(subtitle.text, subtitle.size) : 0,
);
const x = ctx.width * 0.06;
const y = ctx.height * 0.88 - height;
const scale = enter <= 0 ? 0.001 : easeOutBack(enter);
const textX = x + height / 2 + dot / 2 + u * 0.03;
const logo = ctx.logos[0];
const badge = logo
? el("image", {
href: logo,
x: x + height / 2 - dot / 2,
y: y + height / 2 - dot / 2,
width: dot,
height: dot,
preserveAspectRatio: "xMidYMid meet",
})
: el("circle", {
cx: x + height / 2,
cy: y + height / 2,
r: dot / 2,
fill: ctx.kit.colors.accent,
});
const body =
el("rect", {
x,
y,
width,
height,
rx: height / 2,
fill: ctx.kit.colors.background,
}) +
badge +
textEl(title.text, {
x: textX,
y: subtitle ? y + title.size * 1.15 : y + height / 2,
"dominant-baseline": subtitle ? undefined : "central",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 700,
"font-size": title.size,
fill: ctx.kit.colors.text,
}) +
(subtitle
? textEl(subtitle.text, {
x: textX,
y: y + title.size * 1.15 + subtitle.size * 1.3,
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": subtitle.size,
fill: ctx.kit.colors.secondary,
})
: "");
const originX = x;
const originY = y + height / 2;
return svgDocument(
ctx,
el(
"g",
{
opacity: 1 - leave,
transform: `translate(${num(originX)} ${num(originY)}) scale(${num(scale)}) translate(${num(-originX)} ${num(-originY)})`,
},
body,
),
);
},
};

View File

@ -0,0 +1,106 @@
import { easeOutBack, easeOutCubic, progress } from "../easing.js";
import type { MotionTemplate } from "../template.js";
import {
ctaButton,
el,
fitText,
fontStack,
fullBackground,
logoEl,
stringParam,
svgDocument,
textEl,
unit,
} from "../template.js";
export const outroLogo: MotionTemplate = {
info: {
id: "outro_logo",
kind: "outro",
name: "Logo outro",
minDurationMs: 1500,
maxDurationMs: 6000,
params: [{ name: "message", type: "string", required: false }],
},
frame(t, ctx) {
const u = unit(ctx);
const d = ctx.durationMs;
const cx = ctx.width / 2;
const cy = ctx.height * 0.44;
const background = easeOutCubic(progress(t, 0, Math.min(500, d * 0.25)));
const pop = progress(t, d * 0.1, d * 0.4);
const scale = pop <= 0 ? 0 : easeOutBack(pop);
const size = u * 0.3 * scale;
const mark =
ctx.logos.length > 0
? logoEl(ctx, { x: cx - size / 2, y: cy - size / 2, w: size, h: size })
: el("circle", { cx, cy, r: Math.max(size / 2, 0.01), fill: ctx.kit.colors.primary });
const message = stringParam(ctx, "message") ?? ctx.kit.name;
const appear = easeOutCubic(progress(t, d * 0.3, d * 0.55));
const fitted = fitText(message, ctx.width * 0.8, u * 0.07, u * 0.035);
const text = textEl(fitted.text, {
x: cx,
y: cy + u * 0.26 + (1 - appear) * u * 0.03,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.heading, "sans-serif"),
"font-weight": 600,
"font-size": fitted.size,
fill: ctx.kit.colors.text,
opacity: appear,
});
return svgDocument(ctx, fullBackground(ctx, background) + mark + text);
},
};
export const outroSubscribe: MotionTemplate = {
info: {
id: "outro_subscribe",
kind: "outro",
name: "Call to action outro",
minDurationMs: 2000,
maxDurationMs: 8000,
params: [
{ name: "cta", type: "string", required: true },
{ name: "handle", type: "string", required: false },
],
},
frame(t, ctx) {
const u = unit(ctx);
const d = ctx.durationMs;
const cx = ctx.width / 2;
const background = easeOutCubic(progress(t, 0, Math.min(500, d * 0.2)));
const logoAppear = easeOutCubic(progress(t, d * 0.05, d * 0.3));
const size = u * 0.18;
const logo = logoEl(
ctx,
{ x: cx - size / 2, y: ctx.height * 0.18, w: size, h: size },
{ opacity: logoAppear },
);
const pop = progress(t, d * 0.15, d * 0.4);
// A gentle 1 Hz breathing pulse once the button has landed; no brightness change.
const pulse = pop >= 1 ? 1 + 0.03 * Math.sin(((t - d * 0.4) / 1000) * Math.PI * 2) : 1;
const button = ctaButton(
ctx,
stringParam(ctx, "cta") ?? "",
cx,
ctx.height * 0.55,
(pop <= 0 ? 0 : easeOutBack(pop)) * pulse,
);
const handle = stringParam(ctx, "handle");
let handleText = "";
if (handle) {
const appear = easeOutCubic(progress(t, d * 0.35, d * 0.6));
const fitted = fitText(handle, ctx.width * 0.8, u * 0.05, u * 0.03);
handleText = textEl(fitted.text, {
x: cx,
y: ctx.height * 0.55 + u * 0.16,
"text-anchor": "middle",
"font-family": fontStack(ctx.fonts.body, "sans-serif"),
"font-size": fitted.size,
fill: ctx.kit.colors.text,
opacity: appear,
});
}
return svgDocument(ctx, fullBackground(ctx, background) + logo + button + handleText);
},
};

View File

@ -0,0 +1,676 @@
import { randomUUID } from "node:crypto";
import { mkdir, mkdtemp, readFile, realpath, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import type {
JobPublisher,
LinkShortener,
MediaInferenceProvider,
MediaPublisher,
MediaRenderer,
MotionGraphicsRenderer,
} from "@rakazo/adapter-kit";
import type {
BrandKit,
Edl,
FfprobeReport,
GateIssue,
GateReport,
MediaAnalysisKind,
MediaEditPlan,
MediaRender,
PlatformProfile,
} from "@rakazo/contracts";
import { BrandKitSchema, EdlSchema, PlatformProfileSchema } from "@rakazo/contracts";
import type { MediaRenderPatch, MediaRepos, PrismaClient } from "@rakazo/db";
import {
ACTIVE_RENDER_STATUSES,
bootstrapUserSpace,
createDb,
createMediaRepos,
IsolationError,
MediaStateConflictError,
mapMediaEditPlan,
mapMediaRender,
} from "@rakazo/db";
import type { CompileSource } from "@rakazo/media";
import { evaluateGate } from "@rakazo/media";
import { LocalArtifactStore } from "./artifacts.js";
import type {
MediaPreparedSources,
MediaPrivacyService,
MediaRealtimeEvent,
MediaServiceDeps,
} from "./media-context.js";
/*
* Test harness for the render service and job: the repos the render slice
* reads, either in memory or backed by PostgreSQL (VERIFY_DATABASE), plus a
* pass-through privacy service. Everything is synthetic.
*/
export function renderTestProfile(overrides: Record<string, unknown> = {}): PlatformProfile {
return PlatformProfileSchema.parse({
id: "example-vertical-test",
name: "Example vertical test",
kind: "own_site",
reviewedAt: "2026-09-01",
video: {
aspectRatios: ["9:16"],
width: 1080,
height: 1920,
fps: 30,
codec: "h264",
maxBitrateKbps: 8000,
maxFileSizeMb: 500,
},
duration: { hardMinMs: 1000, hardMaxMs: 120_000, hookWindowMs: 2000 },
safeZones: { topPct: 8, bottomPct: 15, leftPct: 4, rightPct: 10 },
captions: { recommended: true, maxLines: 2 },
audio: { targetLufs: -14, truePeakDb: -1 },
text: { captionMaxChars: 500, hashtagLimit: 5, linkPolicy: "allowed" },
content: { thresholds: { violence: 0.5 } },
disclosure: { aiAlteredLabelRequired: true },
consent: { requirePerformerConsent: false, requireAgeVerification: false },
export: {},
...overrides,
});
}
export function renderTestKit(input: {
logoArtifactId: string;
musicArtifactId: string;
}): BrandKit {
return BrandKitSchema.parse({
name: "Studio Example",
colors: {
primary: "#224488",
secondary: "#446688",
accent: "#FFAA00",
text: "#FFFFFF",
background: "#101010",
},
logoArtifactIds: [input.logoArtifactId],
watermark: {
artifactId: input.logoArtifactId,
position: "top_right",
opacity: 0.8,
scale: 0.2,
},
captionPresets: [
{
id: "clean",
name: "Clean",
template: "clean",
fontFamily: "DejaVu Sans",
sizePct: 5,
textColor: "#FFFFFF",
},
],
musicBeds: [
{ artifactId: input.musicArtifactId, title: "Bed", license: "own", commercialUse: true },
],
});
}
export interface HarnessAsset {
assetId: string;
source: CompileSource;
}
export interface RenderHarness {
backend: "memory" | "postgres";
mediaRoot: string;
spaceId: string;
userId: string;
deps: MediaServiceDeps;
events: MediaRealtimeEvent[];
enqueued: string[];
/** Register an asset; the caller writes the file at `assetPath(assetId, fileName)`. */
createAsset(name: string): Promise<string>;
assetPath(assetId: string, fileName: string): string;
setSource(assetId: string, source: CompileSource): void;
sources: Map<string, CompileSource>;
addAnalysis(assetId: string, kind: MediaAnalysisKind, data: unknown): Promise<void>;
addSegments(assetId: string, ranges: Array<{ startMs: number; endMs: number }>): Promise<void>;
addArtifact(file: string, name: string, mimeType: string): Promise<string>;
createProject(input: { assetIds: string[]; kit: BrandKit | null; profileId: string }): Promise<string>;
createPlan(projectId: string, edl: unknown): Promise<MediaEditPlan>;
planVersions(projectId: string, profileId: string): Promise<MediaEditPlan[]>;
artifactCount(): Promise<number>;
cleanup(): Promise<void>;
}
export interface HarnessOptions {
profiles: PlatformProfile[];
renderer: MediaRenderer;
motion: MotionGraphicsRenderer;
inference?: MediaInferenceProvider | null;
now?: () => Date;
}
const unusedPublisher: MediaPublisher = {
id: "unused",
publish: async () => {
throw new Error("not used by render tests");
},
};
/** PostgreSQL when VERIFY_DATABASE and DATABASE_URL are set, otherwise in memory. */
export async function createRenderHarness(options: HarnessOptions): Promise<RenderHarness> {
const dir = await realpath(await mkdtemp(path.join(tmpdir(), "media-render-")));
const mediaRoot = path.join(dir, "media");
await mkdir(mediaRoot, { recursive: true });
const artifacts = new LocalArtifactStore(path.join(dir, "store"));
const events: MediaRealtimeEvent[] = [];
const enqueued: string[] = [];
const jobs: JobPublisher = {
enqueue: async (job) => {
enqueued.push(`${job.name}:${JSON.stringify(job.payload)}`);
},
cancel: async () => undefined,
close: async () => undefined,
};
const databaseUrl = process.env.DATABASE_URL;
const backend =
process.env.VERIFY_DATABASE && databaseUrl
? await postgresBackend(databaseUrl)
: memoryBackend(options.now ?? (() => new Date()));
const sources = new Map<string, CompileSource>();
const deps: MediaServiceDeps = {
prisma: backend.prisma,
repos: backend.repos,
jobs,
inference: options.inference ?? null,
renderer: options.renderer,
motion: options.motion,
publisher: unusedPublisher,
shortener: null as LinkShortener | null,
sources: {},
mediaRoot,
profiles: async () => ({
profiles: new Map(
options.profiles.map((profile) => [profile.id, { profile, origin: "builtin" as const }]),
),
errors: [],
}),
artifacts,
model: { generate: async () => null },
publish: async (_spaceId, event) => {
events.push(event);
},
now: options.now ?? (() => new Date()),
};
const context = (userId: string) => ({
operationId: "test",
traceId: "test",
spaceId: backend.spaceId,
userId,
signal: new AbortController().signal,
});
return {
backend: backend.kind,
mediaRoot,
spaceId: backend.spaceId,
userId: backend.userId,
deps,
events,
enqueued,
sources,
createAsset: (name) => backend.createAsset(name),
assetPath: (assetId, fileName) =>
path.join(mediaRoot, backend.spaceId, "assets", assetId, fileName),
setSource: (assetId, source) => sources.set(assetId, source),
addAnalysis: (assetId, kind, data) => backend.addAnalysis(assetId, kind, data),
addSegments: (assetId, ranges) => backend.addSegments(assetId, ranges),
async addArtifact(file, name, mimeType) {
const bytes = new Uint8Array(await readFile(file));
const stored = await artifacts.put({ name, mimeType, bytes }, context(backend.userId));
const row = await backend.prisma.artifact.create({
data: {
spaceId: backend.spaceId,
userId: backend.userId,
name,
mimeType,
size: bytes.byteLength,
hash: "test",
storageKey: stored.id,
},
select: { id: true },
});
return row.id;
},
createProject: (input) => backend.createProject(input),
createPlan: (projectId, edl) => {
const parsed = EdlSchema.parse(edl);
return backend.repos.createPlanVersion(backend.spaceId, {
projectId,
profileId: parsed.output.profileId,
edl: parsed,
createdBy: "user",
rationale: null,
});
},
planVersions: (projectId, profileId) =>
backend.repos.listPlanVersions(backend.spaceId, projectId, profileId),
artifactCount: () => backend.prisma.artifact.count({ where: { spaceId: backend.spaceId } }),
async cleanup() {
await backend.cleanup();
await rm(dir, { recursive: true, force: true });
},
};
}
interface Backend {
kind: "memory" | "postgres";
prisma: PrismaClient;
repos: MediaRepos;
spaceId: string;
userId: string;
createAsset(name: string): Promise<string>;
addAnalysis(assetId: string, kind: MediaAnalysisKind, data: unknown): Promise<void>;
addSegments(assetId: string, ranges: Array<{ startMs: number; endMs: number }>): Promise<void>;
createProject(input: { assetIds: string[]; kit: BrandKit | null; profileId: string }): Promise<string>;
cleanup(): Promise<void>;
}
async function postgresBackend(databaseUrl: string): Promise<Backend> {
const db = createDb(databaseUrl, { poolMax: 4 });
const prisma = db.prisma;
const repos = createMediaRepos(prisma);
const userId = `mr${randomUUID().replaceAll("-", "")}`;
await prisma.user.create({ data: { id: userId, name: "Test", email: `${userId}@rakazo.test` } });
const { spaceId } = await bootstrapUserSpace(
prisma,
{ id: userId },
{ signupsEnabled: "true", signupAllowlist: "" },
{ claimDeploymentOwner: false },
);
const source = await repos.createSource(spaceId, {
kind: "folder",
label: "Synthetic",
ownerUserId: userId,
config: {},
});
let analysisVersion = 0;
return {
kind: "postgres",
prisma,
repos,
spaceId,
userId,
async createAsset(name) {
const result = await repos.upsertSourceItems(spaceId, source.id, [
{
externalId: name,
kind: "video",
capturedAt: new Date(Date.UTC(2026, 0, 1)),
durationMs: 8000,
width: 640,
height: 360,
fps: 30,
gpsLat: null,
gpsLon: null,
deviceModel: null,
albumNames: [],
isFavorite: false,
fileName: name,
sizeBytes: 1,
revision: "r1",
},
]);
return result.changedAssetIds[0]!;
},
async addAnalysis(assetId, kind, data) {
analysisVersion += 1;
await repos.upsertAnalysis(spaceId, assetId, {
kind,
modelId: "emulator",
modelVersion: `v${analysisVersion}`,
data: data as never,
});
},
async addSegments(assetId, ranges) {
await repos.replaceSegments(
spaceId,
assetId,
1,
ranges.map((range) => ({ kind: "shot", ...range })),
);
},
async createProject(input) {
const kit = input.kit ? await repos.upsertBrandKit(spaceId, { kit: input.kit }) : null;
const project = await repos.createProject(spaceId, userId, {
title: "Synthetic project",
goal: "teaser",
profileIds: [input.profileId],
assetIds: input.assetIds,
brandKitId: kit?.id ?? null,
});
return project.id;
},
async cleanup() {
const memberships = await prisma.member.findMany({
where: { userId },
select: { organizationId: true },
});
await prisma.organization.deleteMany({
where: { id: { in: memberships.map((member) => member.organizationId) } },
});
await prisma.user.deleteMany({ where: { id: userId } });
await prisma.$disconnect();
await db.pool.end();
},
};
}
interface RenderRow {
id: string;
spaceId: string;
projectId: string;
planId: string;
profileId: string;
quality: string;
status: string;
progress: number;
outputs: unknown[];
gateReport: unknown;
ffprobe: unknown;
autoFixRound: number;
error: string | null;
createdBy: string;
createdAt: Date;
updatedAt: Date;
}
/** In-memory stand-in for the repos the render slice uses, with the same state rules. */
function memoryBackend(now: () => Date): Backend {
const spaceId = "space-memory";
const userId = "user-memory";
let counter = 0;
const id = (prefix: string) => `${prefix}${(++counter).toString().padStart(6, "0")}`;
const assets = new Set<string>();
const analyses = new Map<string, unknown>();
const segments: Array<{
id: string;
assetId: string;
startMs: number;
endMs: number;
paidOnly: boolean;
}> = [];
const kits = new Map<string, BrandKit>();
const projects = new Map<
string,
{ id: string; assetIds: string[]; collectionIds: string[]; brandKitId: string | null }
>();
const plans: Array<Parameters<typeof mapMediaEditPlan>[0]> = [];
const renders = new Map<string, RenderRow>();
const artifacts: Array<Record<string, unknown> & { id: string; spaceId: string }> = [];
const requireRender = (space: string, renderId: string) => {
const row = renders.get(renderId);
if (!row || row.spaceId !== space) throw new IsolationError();
return row;
};
const requirePlan = (space: string, planId: string) => {
const row = plans.find((plan) => plan.id === planId && plan.spaceId === space);
if (!row) throw new IsolationError();
return row;
};
const toRender = (row: RenderRow): MediaRender =>
mapMediaRender(structuredClone(row) as unknown as Parameters<typeof mapMediaRender>[0]);
const repos = {
async mediaSpaceOf(kind: string, renderId: string) {
return kind === "render" ? (renders.get(renderId)?.spaceId ?? null) : null;
},
async getPlan(space: string, planId: string) {
return mapMediaEditPlan(requirePlan(space, planId));
},
async createPlanVersion(
space: string,
input: {
projectId: string;
profileId: string;
edl: unknown;
createdBy: "agent" | "user";
rationale: string | null;
},
) {
const edl = EdlSchema.parse(input.edl);
if (edl.output.profileId !== input.profileId) throw new Error("profile mismatch");
if (space !== spaceId || !projects.has(input.projectId)) throw new IsolationError();
const version =
plans
.filter((p) => p.projectId === input.projectId && p.profileId === input.profileId)
.reduce((max, p) => Math.max(max, p.version), 0) + 1;
const row = {
id: id("plan"),
spaceId: space,
projectId: input.projectId,
profileId: input.profileId,
version,
edl: structuredClone(edl),
createdBy: input.createdBy,
rationale: input.rationale,
createdAt: now(),
} as unknown as Parameters<typeof mapMediaEditPlan>[0];
plans.push(row);
return mapMediaEditPlan(row);
},
async listPlanVersions(space: string, projectId: string, profileId: string) {
return plans
.filter((p) => p.spaceId === space && p.projectId === projectId && p.profileId === profileId)
.sort((a, b) => b.version - a.version)
.map(mapMediaEditPlan);
},
async getProjectRecord(space: string, projectId: string) {
const project = projects.get(projectId);
if (!project || space !== spaceId) throw new IsolationError();
return { ...project, createdBy: userId };
},
async getBrandKit(space: string, kitId: string) {
const kit = kits.get(kitId);
if (!kit || space !== spaceId) throw new IsolationError();
return { id: kitId, kit, createdAt: now().toISOString(), updatedAt: now().toISOString() };
},
async collectionAssetIds() {
return [];
},
async listSegments(space: string, assetIds: string[]) {
if (space !== spaceId) return [];
return segments.filter((segment) => assetIds.includes(segment.assetId));
},
async latestAnalysis(space: string, assetId: string, kind: string) {
if (space !== spaceId || !assets.has(assetId)) throw new IsolationError();
const data = analyses.get(`${assetId}:${kind}`);
return data === undefined ? null : { data };
},
async createRender(
space: string,
input: { planId: string; quality: string; createdBy: string },
) {
const plan = requirePlan(space, input.planId);
const row: RenderRow = {
id: id("render"),
spaceId: space,
projectId: plan.projectId,
planId: plan.id,
profileId: plan.profileId,
quality: input.quality,
status: "queued",
progress: 0,
outputs: [],
gateReport: null,
ffprobe: null,
autoFixRound: 0,
error: null,
createdBy: input.createdBy,
createdAt: now(),
updatedAt: now(),
};
renders.set(row.id, row);
return toRender(row);
},
async getRender(space: string, renderId: string) {
return toRender(requireRender(space, renderId));
},
async getRenderRecord(space: string, renderId: string) {
return structuredClone(requireRender(space, renderId));
},
async claimRender(space: string, renderId: string) {
const row = requireRender(space, renderId);
if (row.status !== "queued") return null;
Object.assign(row, { status: "rendering", progress: 0, error: null, updatedAt: now() });
return toRender(row);
},
async updateRender(
space: string,
renderId: string,
patch: MediaRenderPatch,
from = ACTIVE_RENDER_STATUSES,
) {
const row = requireRender(space, renderId);
if (!from.includes(row.status as never)) return null;
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) continue;
(row as unknown as Record<string, unknown>)[key] =
key === "progress" ? Math.min(1, Math.max(0, value as number)) : structuredClone(value);
}
row.updatedAt = now();
return toRender(row);
},
async cancelRender(space: string, renderId: string) {
const row = requireRender(space, renderId);
if (!ACTIVE_RENDER_STATUSES.includes(row.status as never)) {
throw new MediaStateConflictError(`Render is already ${row.status}`);
}
row.status = "canceled";
row.updatedAt = now();
return toRender(row);
},
};
const prisma = {
artifact: {
async findFirst(args: { where: { id: string; spaceId: string } }) {
return (
artifacts.find((a) => a.id === args.where.id && a.spaceId === args.where.spaceId) ?? null
);
},
async create(args: { data: Record<string, unknown> & { spaceId: string } }) {
const row = { ...args.data, id: id("artifact") };
artifacts.push(row);
return row;
},
async deleteMany(args: { where: { id: string; spaceId: string } }) {
const index = artifacts.findIndex(
(a) => a.id === args.where.id && a.spaceId === args.where.spaceId,
);
if (index >= 0) artifacts.splice(index, 1);
return { count: index >= 0 ? 1 : 0 };
},
async count(args: { where: { spaceId: string } }) {
return artifacts.filter((a) => a.spaceId === args.where.spaceId).length;
},
},
};
return {
kind: "memory",
prisma: prisma as unknown as PrismaClient,
repos: repos as unknown as MediaRepos,
spaceId,
userId,
async createAsset() {
const assetId = id("asset");
assets.add(assetId);
return assetId;
},
async addAnalysis(assetId, kind, data) {
analyses.set(`${assetId}:${kind}`, data);
},
async addSegments(assetId, ranges) {
for (const range of ranges) {
segments.push({ id: id("segment"), assetId, ...range, paidOnly: false });
}
},
async createProject(input) {
let brandKitId: string | null = null;
if (input.kit) {
brandKitId = id("kit");
kits.set(brandKitId, input.kit);
}
const projectId = id("project");
projects.set(projectId, {
id: projectId,
assetIds: input.assetIds,
collectionIds: [],
brandKitId,
});
return projectId;
},
async cleanup() {},
};
}
// ---------------------------------------------------------------------------
// Privacy stand-in
// ---------------------------------------------------------------------------
export interface FakePrivacy extends MediaPrivacyService {
gateCalls: Array<{ autoFixRound: number; edl: Edl; ffprobe: FfprobeReport }>;
}
/**
* Pass-through sources (no redactions in these plans) and the deterministic
* gate on the probed output. `failWith` adds a blocking content issue to every
* report, like a classifier hit at the start of the output.
*/
export function createFakePrivacy(
sources: Map<string, CompileSource>,
options: { failWith?: GateIssue; now?: () => Date } = {},
): FakePrivacy {
const gateCalls: FakePrivacy["gateCalls"] = [];
const unsupported = async (): Promise<never> => {
throw new Error("not used by render tests");
};
return {
gateCalls,
async prepareSources(input): Promise<MediaPreparedSources> {
const prepared = new Map<string, CompileSource>();
for (const clip of input.edl.tracks.video) {
const source = sources.get(clip.assetId);
if (!source) throw new Error(`no source for ${clip.assetId}`);
prepared.set(clip.assetId, source);
}
return { sources: prepared, aiAlterations: [], fallbacks: [] };
},
async gate(input): Promise<GateReport> {
gateCalls.push({ autoFixRound: input.autoFixRound, edl: input.edl, ffprobe: input.ffprobe });
const report = evaluateGate({
profile: input.profile,
edl: input.edl,
ffprobe: input.ffprobe,
content: [],
faces: [],
trackingGaps: [],
marks: [],
people: [],
copy: input.copy,
aiAlterations: [],
bundleHasDisclosure: input.bundleHasDisclosure,
watermarkBox: null,
sampledFrames: 0,
autoFixRound: input.autoFixRound,
now: (options.now ?? (() => new Date()))(),
});
if (!options.failWith) return report;
return { ...report, passed: false, issues: [...report.issues, options.failWith] };
},
createMark: unsupported,
updateMark: unsupported,
removePerson: unsupported,
};
}

View File

@ -0,0 +1,835 @@
import { createHash } from "node:crypto";
import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
import path from "node:path";
import type { AdapterContext, MediaJobHandlers, MediaRenderProgress } from "@rakazo/adapter-kit";
import { mediaRenderJob, UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import type {
AiAlteration,
BrandKit,
Edl,
GateReport,
MediaRender,
MediaRenderOutput,
MediaRenderQuality,
PlatformProfile,
RenderPlan,
} from "@rakazo/contracts";
import { MAX_AUTO_FIX_ROUNDS } from "@rakazo/contracts";
import type { MediaRenderPatch } from "@rakazo/db";
import { InvalidMediaInputError } from "@rakazo/db";
import { getLogger } from "@rakazo/logging";
import type { CaptionWord } from "@rakazo/media";
import { compileRenderPlan, findMetadataLeaks, planAutoFix } from "@rakazo/media";
import { resolveInside } from "@rakazo/media/node";
import type {
MediaActor,
MediaPrivacyService,
MediaRenderService,
MediaServiceDeps,
} from "./media-context.js";
import { MediaToolMissingError } from "./media-folder-source-probe.js";
/*
* Render service (API side) and render job (worker side).
*
* A render runs: claim → prepare redacted sources (privacy slice) → pre-render
* motion clips → compile → ffmpeg → probe → gate. A blocked gate gets up to
* MAX_AUTO_FIX_ROUNDS compliance-only repairs (planAutoFix); each repair is
* stored as a new plan version by the agent and rendered again under the same
* render id. Only a passed gate moves outputs out of the work directory and
* marks the render done; every other outcome removes the render's files.
*
* Layout: `<spaceId>/renders/<renderId>/{video.mp4,poster.jpg,captions.srt,
* captions.vtt}` with intermediates in `work/` beside them.
*
* Cancel: the API marks the render canceled and aborts it when this process
* runs it. A worker in another process notices through a failed status update
* (updateRender returns null once canceled) or the periodic status check.
*/
/** Renders running in this process, so a cancel can kill ffmpeg immediately. */
export type RunningMediaRenders = Map<string, AbortController>;
const runningRenders: RunningMediaRenders = new Map();
const PROGRESS_INTERVAL_MS = 1000;
const CANCEL_POLL_MS = 2000;
/** Share of the progress bar spent rendering; probing and gating take the rest. */
const RENDER_SHARE = 0.85;
const OUTPUT_FILES = {
video: "video.mp4",
poster: "poster.jpg",
subtitles_srt: "captions.srt",
subtitles_vtt: "captions.vtt",
} as const;
const OUTPUT_MIME: Record<keyof typeof OUTPUT_FILES, string> = {
video: "video/mp4",
poster: "image/jpeg",
subtitles_srt: "application/x-subrip",
subtitles_vtt: "text/vtt",
};
const EXTENSION_BY_MIME: Record<string, string> = {
"image/png": ".png",
"image/jpeg": ".jpg",
"image/svg+xml": ".svg",
"image/webp": ".webp",
"audio/mpeg": ".mp3",
"audio/wav": ".wav",
"audio/x-wav": ".wav",
"audio/aac": ".aac",
"audio/mp4": ".m4a",
"audio/ogg": ".ogg",
"audio/flac": ".flac",
"font/ttf": ".ttf",
"font/otf": ".otf",
"font/collection": ".ttc",
};
/** A failure whose message is safe to show the user (no paths, no tool output). */
export class MediaRenderFailure extends Error {
constructor(message: string) {
super(message);
this.name = "MediaRenderFailure";
}
}
class RenderCanceled extends Error {
constructor() {
super("render canceled");
this.name = "RenderCanceled";
}
}
function userMessage(error: unknown): string {
if (error instanceof MediaRenderFailure) return error.message;
if (error instanceof UnsupportedMediaOperationError) {
return `${error.operation} is not available on this server.`;
}
if (error instanceof MediaToolMissingError) return "ffmpeg is not installed on the media worker.";
return "Rendering failed. Check the plan and try again.";
}
function renderEvent(render: MediaRender) {
return {
type: "media.render" as const,
renderId: render.id,
projectId: render.projectId,
status: render.status,
progress: render.progress,
};
}
async function publishRender(deps: MediaServiceDeps, spaceId: string, render: MediaRender) {
try {
await deps.publish(spaceId, renderEvent(render));
} catch (error) {
getLogger().warn("media render event not published", {
renderId: render.id,
reason: error instanceof Error ? error.message : String(error),
});
}
}
// ---------------------------------------------------------------------------
// Service
// ---------------------------------------------------------------------------
export function createMediaRenderService(
deps: MediaServiceDeps,
options: { running?: RunningMediaRenders } = {},
): MediaRenderService {
const running = options.running ?? runningRenders;
return {
async start(actor, input) {
const plan = await deps.repos.getPlan(actor.spaceId, input.planId);
const { profiles } = await deps.profiles();
if (!profiles.has(plan.profileId)) {
throw new InvalidMediaInputError("The plan's platform profile is not available");
}
const render = await deps.repos.createRender(actor.spaceId, {
planId: plan.id,
quality: input.quality,
createdBy: actor.userId,
});
await deps.jobs.enqueue(mediaRenderJob(render.id));
await publishRender(deps, actor.spaceId, render);
return render;
},
async cancel(actor, renderId) {
const render = await deps.repos.cancelRender(actor.spaceId, renderId);
running.get(renderId)?.abort();
await deps.jobs.cancel(mediaRenderJob(renderId).replaceKey!).catch((error: unknown) => {
getLogger().warn("media render job not dequeued", {
renderId,
reason: error instanceof Error ? error.message : String(error),
});
});
await publishRender(deps, actor.spaceId, render);
return render;
},
async outputPath(actor, renderId, fileName) {
const record = await deps.repos.getRenderRecord(actor.spaceId, renderId);
if (record.status !== "done") return null;
if (path.basename(fileName) !== fileName || fileName.startsWith(".")) return null;
const outputs = Array.isArray(record.outputs) ? record.outputs : [];
const recorded = outputs.some(
(output) =>
output !== null &&
typeof output === "object" &&
!Array.isArray(output) &&
output.fileName === fileName,
);
if (!recorded) return null;
// Derive the location instead of trusting a stored path.
const file = await resolveInside(
deps.mediaRoot,
`${actor.spaceId}/renders/${renderId}/${fileName}`,
);
const found = await stat(file).catch(() => null);
return found?.isFile() ? file : null;
},
};
}
// ---------------------------------------------------------------------------
// Job handler
// ---------------------------------------------------------------------------
interface BrandFiles {
/** Font artifact id → file. */
fonts: Map<string, string>;
fontList: Array<{ family: string; path: string }>;
logos: string[];
watermark: string | null;
lut: string | null;
}
function safeExtension(name: string, mimeType: string): string {
const fromName = path.extname(name).toLowerCase();
if (/^\.[a-z0-9]{1,5}$/.test(fromName)) return fromName;
return EXTENSION_BY_MIME[mimeType.toLowerCase()] ?? ".bin";
}
const even = (value: number) => Math.max(2, Math.floor(value / 2) * 2);
function finiteNumber(value: unknown): value is number {
return typeof value === "number" && Number.isFinite(value);
}
/** Transcript words from a stored TranscribeResult. */
export function transcriptWords(data: unknown): CaptionWord[] {
const words = (data as { words?: unknown } | null)?.words;
if (!Array.isArray(words)) return [];
return words.flatMap((word) => {
const w = word as { startMs?: unknown; endMs?: unknown; text?: unknown };
if (!finiteNumber(w.startMs) || !finiteNumber(w.endMs) || typeof w.text !== "string") {
return [];
}
return [{ startMs: w.startMs, endMs: w.endMs, text: w.text }];
});
}
type Subject = { tMs: number; cx: number; cy: number; strength: number };
/** Reframe subjects from a stored saliency result, or face centres from detections. */
export function subjectPoints(saliency: unknown, faces: unknown): Subject[] {
const points = (saliency as { points?: unknown } | null)?.points;
if (Array.isArray(points)) {
const parsed = points.flatMap((point) => {
const p = point as Partial<Subject>;
return finiteNumber(p.tMs) &&
finiteNumber(p.cx) &&
finiteNumber(p.cy) &&
finiteNumber(p.strength)
? [{ tMs: p.tMs, cx: p.cx, cy: p.cy, strength: p.strength }]
: [];
});
if (parsed.length > 0) return parsed;
}
const frames = (faces as { frames?: unknown } | null)?.frames;
if (!Array.isArray(frames)) return [];
return frames.flatMap((frame) => {
const f = frame as { tMs?: unknown; faces?: unknown };
if (!finiteNumber(f.tMs) || !Array.isArray(f.faces)) return [];
const tMs = f.tMs;
return f.faces.flatMap((face) => {
const box = (face as { box?: { x?: unknown; y?: unknown; w?: unknown; h?: unknown } }).box;
const confidence = (face as { confidence?: unknown }).confidence;
if (!box || !finiteNumber(box.x) || !finiteNumber(box.y)) return [];
if (!finiteNumber(box.w) || !finiteNumber(box.h)) return [];
return [
{
tMs,
cx: box.x + box.w / 2,
cy: box.y + box.h / 2,
strength: finiteNumber(confidence) ? confidence : 1,
},
];
});
});
}
export interface MediaRenderHandlerOptions {
running?: RunningMediaRenders;
/** How often a running render checks whether another process canceled it. */
cancelPollMs?: number;
}
/**
* Worker handler for `media.render`. Never throws for a render failure: the
* render record carries the outcome, so a job retry has nothing to redo.
*/
export function createMediaRenderHandler(
deps: MediaServiceDeps,
privacy: MediaPrivacyService,
options: MediaRenderHandlerOptions = {},
): MediaJobHandlers["media.render"] {
const running = options.running ?? runningRenders;
const pollMs = options.cancelPollMs ?? CANCEL_POLL_MS;
const { repos } = deps;
return async ({ renderId }) => {
const spaceId = await repos.mediaSpaceOf("render", renderId);
if (!spaceId) return;
const claimed = await repos.claimRender(spaceId, renderId);
if (!claimed) return;
await publishRender(deps, spaceId, claimed);
const controller = new AbortController();
running.set(renderId, controller);
const poll = setInterval(() => {
repos
.getRender(spaceId, renderId)
.then((render) => {
if (render.status === "canceled") controller.abort();
})
.catch(() => undefined);
}, pollMs);
poll.unref?.();
const createdArtifacts: Array<{ id: string; storageKey: string }> = [];
let renderDir: string | null = null;
let userId = "";
try {
renderDir = await resolveInside(deps.mediaRoot, `${spaceId}/renders/${renderId}`);
const record = await repos.getRenderRecord(spaceId, renderId);
userId = record.createdBy;
await runRender({
deps,
privacy,
spaceId,
renderId,
renderDir,
record,
signal: controller.signal,
abort: () => controller.abort(),
createdArtifacts,
});
} catch (error) {
if (renderDir) await rm(renderDir, { recursive: true, force: true }).catch(() => undefined);
await removeArtifacts(deps, spaceId, userId, renderId, createdArtifacts);
const current = await repos.getRender(spaceId, renderId).catch(() => null);
if (current?.status === "canceled") {
await publishRender(deps, spaceId, current);
return;
}
getLogger().error("media render failed", {
renderId,
reason: error instanceof Error ? error.message : String(error),
});
const failed = await repos
.updateRender(spaceId, renderId, { status: "failed", error: userMessage(error) })
.catch(() => null);
if (failed) await publishRender(deps, spaceId, failed);
} finally {
clearInterval(poll);
running.delete(renderId);
}
};
}
function artifactContext(spaceId: string, userId: string, renderId: string): AdapterContext {
return {
operationId: `media-render:${renderId}`,
traceId: `media-render:${renderId}`,
spaceId,
userId,
signal: new AbortController().signal,
};
}
async function removeArtifacts(
deps: MediaServiceDeps,
spaceId: string,
userId: string,
renderId: string,
created: Array<{ id: string; storageKey: string }>,
) {
for (const artifact of created) {
await deps.prisma.artifact
.deleteMany({ where: { id: artifact.id, spaceId } })
.catch(() => undefined);
await deps.artifacts
.remove(artifact.storageKey, artifactContext(spaceId, userId, renderId))
.catch(() => undefined);
}
}
type RenderRecord = Awaited<ReturnType<MediaServiceDeps["repos"]["getRenderRecord"]>>;
interface RunInput {
deps: MediaServiceDeps;
privacy: MediaPrivacyService;
spaceId: string;
renderId: string;
renderDir: string;
record: RenderRecord;
signal: AbortSignal;
abort: () => void;
createdArtifacts: Array<{ id: string; storageKey: string }>;
}
async function runRender(input: RunInput): Promise<void> {
const { deps, privacy, spaceId, renderId, renderDir, record, signal } = input;
const { repos } = deps;
const actor: MediaActor = { spaceId, userId: record.createdBy };
const context = artifactContext(spaceId, actor.userId, renderId);
/** Apply a patch while the render is active; a null result means it was canceled. */
const update = async (patch: MediaRenderPatch) => {
const updated = await repos.updateRender(spaceId, renderId, patch);
if (!updated) {
input.abort();
throw new RenderCanceled();
}
return updated;
};
const checkCanceled = () => {
if (signal.aborted) throw new RenderCanceled();
};
let plan = await repos.getPlan(spaceId, record.planId);
const project = await repos.getProjectRecord(spaceId, record.projectId);
const entry = (await deps.profiles()).profiles.get(record.profileId);
if (!entry) throw new MediaRenderFailure("The platform profile is no longer available.");
const profile = entry.profile;
const kit = project.brandKitId
? (await repos.getBrandKit(spaceId, project.brandKitId)).kit
: null;
const quality = record.quality as MediaRenderQuality;
// A retried job starts from a clean work directory.
const workDir = path.join(renderDir, "work");
await rm(renderDir, { recursive: true, force: true });
await mkdir(workDir, { recursive: true });
const fetchArtifact = async (artifactId: string, dir: string, baseName: string) => {
const row = await deps.prisma.artifact.findFirst({
where: { id: artifactId, spaceId },
select: { storageKey: true, name: true, mimeType: true },
});
if (!row) return null;
const bytes = await deps.artifacts.get(row.storageKey, context);
await mkdir(dir, { recursive: true });
const file = path.join(dir, `${baseName}${safeExtension(row.name, row.mimeType)}`);
await writeFile(file, bytes);
return file;
};
const brand = await loadBrandFiles(kit, path.join(workDir, "brand"), fetchArtifact);
const audioFiles = new Map<string, string>();
const alternatives = await alternativeSegments(deps, spaceId, project, plan.edl);
let round = record.autoFixRound;
let edl = plan.edl;
for (;;) {
checkCanceled();
const roundDir = path.join(workDir, `round-${round}`);
await mkdir(roundDir, { recursive: true });
const prepared = await privacy.prepareSources({
actor,
projectId: project.id,
edl,
quality,
workDir: roundDir,
signal,
});
checkCanceled();
for (const clip of edl.tracks.audio) {
if (audioFiles.has(clip.artifactId)) continue;
const file = await fetchArtifact(
clip.artifactId,
path.join(workDir, "audio"),
`audio-${audioFiles.size + 1}`,
);
if (!file) throw new MediaRenderFailure("An audio file of the plan is no longer available.");
audioFiles.set(clip.artifactId, file);
}
const motionClips = await renderMotionClips(deps, edl, kit, brand, roundDir, signal);
const { transcripts, subjects } = await analysisInputs(deps, spaceId, edl);
const videoEncoder = await pickEncoder(deps, profile, quality);
const renderPlan = compileRenderPlan({
edl,
profile,
brandKit: kit,
quality,
workDir: roundDir,
sources: prepared.sources,
transcripts,
subjects,
motionClips,
files: {
lutPath: brand.lut,
watermarkPath: brand.watermark,
fonts: brand.fonts,
audio: audioFiles,
logoPath: brand.logos[0] ?? null,
},
videoEncoder,
});
await executePlan(input, renderPlan, update);
const video = renderPlan.outputs.find((output) => output.kind === "video");
if (!video) throw new Error("render plan has no video output");
const ffprobe = await deps.renderer.probe(video.path, { measureLoudness: true, signal });
if (findMetadataLeaks(ffprobe.tags).length > 0) {
// Spec 5: a render whose metadata could not be removed is failed, not gated.
throw new MediaRenderFailure("The export still carried source metadata.");
}
const gating = await update({ status: "gating", progress: 0.9, ffprobe });
await publishRender(deps, spaceId, gating);
const aiAlterations = [
...new Set<AiAlteration>([...edl.metadata.aiAlterations, ...prepared.aiAlterations]),
];
const rendered: Edl = {
...edl,
metadata: {
...edl.metadata,
aiAlterations,
aiAltered: edl.metadata.aiAltered || aiAlterations.length > 0,
},
};
const report = await privacy.gate({
actor,
projectId: project.id,
edl: rendered,
profile,
outputPath: video.path,
ffprobe,
copy: {
title: edl.metadata.title ?? null,
caption: edl.metadata.description ?? null,
tags: edl.metadata.tags,
},
// The publish bundle always carries the AI disclosure file and checklist
// entry when the profile requires a label and the output is altered.
bundleHasDisclosure: profile.disclosure.aiAlteredLabelRequired && aiAlterations.length > 0,
autoFixRound: round,
signal,
});
checkCanceled();
if (report.passed) {
await finish(input, renderPlan, report, plan.id, update, context);
return;
}
const fix =
round < MAX_AUTO_FIX_ROUNDS
? planAutoFix(edl, { ...report, autoFixRound: round }, alternatives)
: null;
if (!fix) {
await rm(renderDir, { recursive: true, force: true });
const blocked = await update({ status: "blocked", progress: 1, gateReport: report });
await publishRender(deps, spaceId, blocked);
return;
}
const codes = [
...new Set(report.issues.filter((i) => i.severity === "block").map((i) => i.code)),
];
plan = await repos.createPlanVersion(spaceId, {
projectId: project.id,
profileId: plan.profileId,
edl: fix.edl,
createdBy: "agent",
rationale: `Auto-fix ${round + 1}/${MAX_AUTO_FIX_ROUNDS} after the gate blocked (${codes.join(
", ",
)}): ${fix.changes.join("; ")}`,
});
round += 1;
edl = plan.edl;
await rm(roundDir, { recursive: true, force: true });
const retry = await update({
status: "rendering",
progress: 0,
autoFixRound: round,
gateReport: report,
ffprobe: null,
});
await publishRender(deps, spaceId, retry);
}
}
async function loadBrandFiles(
kit: BrandKit | null,
dir: string,
fetchArtifact: (artifactId: string, dir: string, baseName: string) => Promise<string | null>,
): Promise<BrandFiles> {
const files: BrandFiles = {
fonts: new Map(),
fontList: [],
logos: [],
watermark: null,
lut: null,
};
if (!kit) return files;
// Missing files are skipped here; the compiler fails only if the plan needs one.
const missing = (what: string) => getLogger().warn("brand kit file unavailable", { what });
for (const [index, font] of kit.fonts.entries()) {
const file = await fetchArtifact(font.artifactId, dir, `font-${index + 1}`);
if (!file) {
missing("font");
continue;
}
files.fonts.set(font.artifactId, file);
files.fontList.push({ family: font.family, path: file });
}
for (const [index, logo] of kit.logoArtifactIds.entries()) {
const file = await fetchArtifact(logo, dir, `logo-${index + 1}`);
if (file) files.logos.push(file);
else missing("logo");
}
const watermarkId = kit.watermark?.artifactId;
if (watermarkId) {
files.watermark = await fetchArtifact(watermarkId, dir, "watermark");
if (!files.watermark) missing("watermark");
} else if (kit.watermark) {
files.watermark = files.logos[0] ?? null;
}
if (kit.lutArtifactId) {
files.lut = await fetchArtifact(kit.lutArtifactId, dir, "look");
if (!files.lut) missing("lut");
}
return files;
}
async function renderMotionClips(
deps: MediaServiceDeps,
edl: Edl,
kit: BrandKit | null,
brand: BrandFiles,
dir: string,
signal: AbortSignal,
): Promise<Map<string, string>> {
const W = edl.output.width;
const H = edl.output.height;
const jobs: Array<{ key: string; template: NonNullable<Edl["intro"]>; w: number; h: number }> =
[];
if (edl.intro && edl.intro.durationMs > 0)
jobs.push({ key: "intro", template: edl.intro, w: W, h: H });
if (edl.outro && edl.outro.durationMs > 0)
jobs.push({ key: "outro", template: edl.outro, w: W, h: H });
for (const overlay of edl.overlays) {
if (!overlay.motion) continue;
const box = overlay.box;
jobs.push({
key: overlay.id,
template: overlay.motion,
w: box ? even(box.w * W) : W,
h: box ? even(box.h * H) : H,
});
}
const clips = new Map<string, string>();
if (jobs.length === 0) return clips;
if (!kit) throw new MediaRenderFailure("Motion templates need a brand kit on the project.");
for (const job of jobs) {
// Keys are SafeId values ([a-z0-9_-]), so they are safe file names.
const result = await deps.motion.render({
template: job.template,
brandKit: kit,
assets: { fonts: brand.fontList, logoPaths: brand.logos },
width: job.w,
height: job.h,
fps: edl.output.fps,
outPath: path.join(dir, `motion-${job.key}.mov`),
signal,
});
clips.set(job.key, result.path);
}
return clips;
}
async function analysisInputs(deps: MediaServiceDeps, spaceId: string, edl: Edl) {
const transcripts = new Map<string, CaptionWord[]>();
const subjects = new Map<string, Subject[]>();
const assetIds = [...new Set(edl.tracks.video.map((clip) => clip.assetId))];
for (const assetId of assetIds) {
const [transcript, saliency, faces] = await Promise.all([
deps.repos.latestAnalysis(spaceId, assetId, "transcript"),
deps.repos.latestAnalysis(spaceId, assetId, "saliency"),
deps.repos.latestAnalysis(spaceId, assetId, "faces"),
]);
const words = transcriptWords(transcript?.data ?? null);
if (words.length > 0) transcripts.set(assetId, words);
const points = subjectPoints(saliency?.data ?? null, faces?.data ?? null);
if (points.length > 0) subjects.set(assetId, points);
}
return { transcripts, subjects };
}
/** Final renders prefer a working hardware encoder of the profile's codec; drafts use software. */
async function pickEncoder(
deps: MediaServiceDeps,
profile: PlatformProfile,
quality: MediaRenderQuality,
): Promise<string> {
const family = profile.video.codec;
const software = family === "h264" ? "libx264" : "libx265";
if (quality === "draft") return software;
const hardware = await deps.renderer.hardwareEncoders();
return hardware.find((encoder) => encoder.startsWith(`${family}_`)) ?? software;
}
/** Segments the auto-fix may swap in: the project's material, never paid-only. */
async function alternativeSegments(
deps: MediaServiceDeps,
spaceId: string,
project: { assetIds: string[]; collectionIds: string[] },
edl: Edl,
) {
const fromCollections = await deps.repos.collectionAssetIds(spaceId, project.collectionIds);
const assetIds = [
...new Set([
...project.assetIds,
...fromCollections,
...edl.tracks.video.map((clip) => clip.assetId),
]),
];
const segments = await deps.repos.listSegments(spaceId, assetIds);
return segments
.filter((segment) => !segment.paidOnly)
.map((segment) => ({
segmentId: segment.id,
assetId: segment.assetId,
startMs: segment.startMs,
endMs: segment.endMs,
}));
}
async function executePlan(
input: RunInput,
plan: RenderPlan,
update: (patch: MediaRenderPatch) => Promise<MediaRender>,
) {
const { deps, spaceId, signal } = input;
let lastWrite = Number.NEGATIVE_INFINITY;
let writing: Promise<void> | null = null;
let failure: unknown = null;
const onProgress = (progress: MediaRenderProgress) => {
const now = deps.now().getTime();
if (writing || now - lastWrite < PROGRESS_INTERVAL_MS) return;
lastWrite = now;
writing = update({ progress: progress.progress * RENDER_SHARE })
.then((render) => publishRender(deps, spaceId, render))
.catch((error: unknown) => {
failure ??= error;
})
.finally(() => {
writing = null;
});
};
try {
await deps.renderer.render(plan, { signal, onProgress });
} finally {
await writing;
}
if (failure) throw failure;
if (signal.aborted) throw new RenderCanceled();
}
async function finish(
input: RunInput,
plan: RenderPlan,
report: GateReport,
planId: string,
update: (patch: MediaRenderPatch) => Promise<MediaRender>,
context: AdapterContext,
) {
const { deps, spaceId, renderId, renderDir } = input;
const outputs: Array<MediaRenderOutput & { path: string; planId: string }> = [];
for (const output of plan.outputs) {
const fileName = OUTPUT_FILES[output.kind];
const relative = `${spaceId}/renders/${renderId}/${fileName}`;
const destination = await resolveInside(deps.mediaRoot, relative);
await rename(output.path, destination);
const { size } = await stat(destination);
let artifactId: string | null = null;
if (output.kind !== "video") {
artifactId = await registerArtifact(input, context, {
name: fileName,
mimeType: OUTPUT_MIME[output.kind],
file: destination,
});
}
outputs.push({
kind: output.kind,
artifactId,
fileName,
sizeBytes: size,
path: relative,
planId,
});
}
const done = await update({
status: "done",
progress: 1,
outputs,
gateReport: report,
error: null,
});
await rm(path.join(renderDir, "work"), { recursive: true, force: true });
await publishRender(deps, spaceId, done);
}
async function registerArtifact(
input: RunInput,
context: AdapterContext,
file: { name: string; mimeType: string; file: string },
): Promise<string> {
const { deps, spaceId } = input;
const bytes = new Uint8Array(await readFile(file.file));
const stored = await deps.artifacts.put(
{ name: file.name, mimeType: file.mimeType, bytes },
context,
);
try {
const row = await deps.prisma.artifact.create({
data: {
spaceId,
userId: context.userId,
name: file.name,
mimeType: file.mimeType,
size: bytes.byteLength,
hash: createHash("sha256").update(bytes).digest("hex"),
storageKey: stored.id,
},
select: { id: true },
});
input.createdArtifacts.push({ id: row.id, storageKey: stored.id });
return row.id;
} catch (error) {
await deps.artifacts.remove(stored.id, context).catch(() => undefined);
throw error;
}
}

View File

@ -0,0 +1,222 @@
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import { MediaInferenceOperationSchema } from "@rakazo/contracts";
import { describe, expect, it } from "vitest";
import { MediaSidecarError, SidecarMediaInferenceProvider } from "./media-sidecar.js";
const TOKEN = "test-token-0123456789abcdef0123456789abcdef";
const SCENES = {
shots: [{ startMs: 0, endMs: 1000, score: 1 }],
scenes: [{ startMs: 0, endMs: 1000 }],
};
interface Call {
url: string;
init: RequestInit;
}
function json(status: number, body: unknown): Response {
return new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json" },
});
}
function fakeFetch(responses: Array<Response | ((init: RequestInit) => Promise<Response>)>) {
const calls: Call[] = [];
const impl = (async (input: string | URL | Request, init?: RequestInit) => {
calls.push({ url: String(input), init: init ?? {} });
const next = responses.shift();
if (!next) throw new Error("unexpected request");
return typeof next === "function" ? next(init ?? {}) : next;
}) as typeof fetch;
return { impl, calls };
}
function client(fetchImpl: typeof fetch, extra: { retryBaseMs?: number } = {}) {
return new SidecarMediaInferenceProvider({
baseUrl: "http://127.0.0.1:8800/",
token: TOKEN,
fetch: fetchImpl,
retryBaseMs: 0,
...extra,
});
}
describe("SidecarMediaInferenceProvider", () => {
it("posts camelCase JSON with bearer auth and without the signal", async () => {
const { impl, calls } = fakeFetch([json(200, SCENES)]);
const controller = new AbortController();
const result = await client(impl).detectScenes({
path: "/data/media/a.mp4",
signal: controller.signal,
});
expect(result).toEqual(SCENES);
expect(calls).toHaveLength(1);
const [call] = calls;
expect(call?.url).toBe("http://127.0.0.1:8800/v1/detectScenes");
expect(call?.init.method).toBe("POST");
const headers = call?.init.headers as Record<string, string>;
expect(headers.authorization).toBe(`Bearer ${TOKEN}`);
expect(headers["content-type"]).toBe("application/json");
expect(JSON.parse(String(call?.init.body))).toEqual({ path: "/data/media/a.mp4" });
expect(call?.init.signal).toBeInstanceOf(AbortSignal);
});
it("reads capabilities with GET", async () => {
const capabilities = {
configured: true,
operations: Object.fromEntries(
MediaInferenceOperationSchema.options.map((operation) => [
operation,
{
available: operation !== "synthesizeIdentity",
experimental: false,
model: "m",
license: "Apache-2.0",
commercialUse: true,
},
]),
),
accelerator: "cpu",
hardwareEncoders: [],
};
const { impl, calls } = fakeFetch([json(200, capabilities)]);
await expect(client(impl).capabilities()).resolves.toEqual(capabilities);
expect(calls[0]?.url).toBe("http://127.0.0.1:8800/v1/capabilities");
expect(calls[0]?.init.method).toBe("GET");
});
it("maps 501 to UnsupportedMediaOperationError", async () => {
const { impl } = fakeFetch([
json(501, {
error: { code: "unsupported", message: "no licensed model", operation: "inpaintVideo" },
}),
]);
const error = await client(impl)
.inpaintVideo({ path: "/m/a.mp4", maskDir: "/m/masks", outPath: "/m/out.mp4" })
.catch((caught: unknown) => caught);
expect(error).toBeInstanceOf(UnsupportedMediaOperationError);
expect((error as UnsupportedMediaOperationError).operation).toBe("inpaintVideo");
expect((error as Error).message).toContain("no licensed model");
});
it("retries 503 busy with backoff, at most three times", async () => {
const busy = () => json(503, { error: { code: "busy", message: "queue full" } });
const recovered = fakeFetch([busy(), busy(), json(200, SCENES)]);
await expect(client(recovered.impl).detectScenes({ path: "/m/a.mp4" })).resolves.toEqual(
SCENES,
);
expect(recovered.calls).toHaveLength(3);
const exhausted = fakeFetch([busy(), busy(), busy(), busy(), busy()]);
const error = await client(exhausted.impl)
.detectScenes({ path: "/m/a.mp4" })
.catch((caught: unknown) => caught);
expect(exhausted.calls).toHaveLength(4);
expect(error).toBeInstanceOf(MediaSidecarError);
expect((error as MediaSidecarError).code).toBe("busy");
expect((error as MediaSidecarError).status).toBe(503);
});
it("maps typed sidecar errors", async () => {
const { impl } = fakeFetch([
json(400, { error: { code: "invalid_path", message: "path is outside MEDIA_ROOT" } }),
json(500, "not json-shaped"),
]);
const provider = client(impl);
await expect(provider.detectScenes({ path: "/etc/passwd" })).rejects.toMatchObject({
name: "MediaSidecarError",
code: "invalid_path",
status: 400,
operation: "detectScenes",
});
await expect(provider.detectScenes({ path: "/m/a.mp4" })).rejects.toMatchObject({
code: "http_error",
status: 500,
});
});
it("passes caller aborts through as AbortError", async () => {
const controller = new AbortController();
const { impl } = fakeFetch([
(init) =>
new Promise<Response>((_, reject) => {
init.signal?.addEventListener("abort", () => reject(init.signal?.reason));
setTimeout(() => controller.abort(), 10);
}),
]);
await expect(
client(impl).trackFaces({ path: "/m/a.mp4", sampleFps: 5, signal: controller.signal }),
).rejects.toMatchObject({ name: "AbortError" });
const already = new AbortController();
already.abort();
const idle = fakeFetch([]);
await expect(
client(idle.impl).detectScenes({ path: "/m/a.mp4", signal: already.signal }),
).rejects.toMatchObject({ name: "AbortError" });
expect(idle.calls).toHaveLength(0);
});
it("reports a timeout as a typed error", async () => {
const { impl } = fakeFetch([
(init) =>
new Promise<Response>((_, reject) => {
init.signal?.addEventListener("abort", () => reject(init.signal?.reason));
}),
]);
const provider = new SidecarMediaInferenceProvider({
baseUrl: "http://127.0.0.1:8800",
token: TOKEN,
fetch: impl,
timeoutsMs: { detectScenes: 20 },
});
await expect(provider.detectScenes({ path: "/m/a.mp4" })).rejects.toMatchObject({
code: "timeout",
});
});
it("rejects malformed responses", async () => {
const { impl } = fakeFetch([
json(200, { shots: [{ startMs: -1, endMs: 10, score: 1 }], scenes: [] }),
json(200, {
frames: [{ tMs: 0, faces: [{ box: { x: 2, y: 0, w: 0.1, h: 0.1 }, confidence: 1 }] }],
}),
json(200, { model: "m", dim: 3, vectors: [[1, 2]] }),
new Response("<html>", { status: 200 }),
]);
const provider = client(impl);
await expect(provider.detectScenes({ path: "/m/a.mp4" })).rejects.toMatchObject({
code: "malformed_response",
});
await expect(provider.detectFaces({ path: "/m/a.mp4", sampleFps: 1 })).rejects.toMatchObject({
code: "malformed_response",
});
await expect(provider.embedFrames({ path: "/m/a.mp4", timesMs: [0] })).rejects.toMatchObject({
code: "malformed_response",
});
await expect(provider.detectScenes({ path: "/m/a.mp4" })).rejects.toMatchObject({
code: "malformed_response",
});
});
it("never puts the token into error messages", async () => {
const { impl } = fakeFetch([
json(401, { error: { code: "unauthorized", message: `bad token ${TOKEN}` } }),
json(500, { error: { code: "internal", message: `crashed with Bearer ${TOKEN}` } }),
async () => {
throw new TypeError(`connect ECONNREFUSED (auth ${TOKEN})`);
},
]);
const provider = client(impl);
for (let i = 0; i < 3; i++) {
const error = (await provider
.detectScenes({ path: "/m/a.mp4" })
.catch((caught: unknown) => caught)) as Error;
expect(error).toBeInstanceOf(MediaSidecarError);
expect(error.message).not.toContain(TOKEN);
expect(JSON.stringify(error)).not.toContain(TOKEN);
expect(String(error.stack)).not.toContain(TOKEN);
}
});
});

View File

@ -0,0 +1,410 @@
import type {
AudioFeaturesResult,
ClassifyContentResult,
DetectFacesResult,
DetectScenesResult,
EmbedFramesResult,
FaceLandmarksResult,
InpaintVideoResult,
MediaInferenceProvider,
SaliencyResult,
SegmentPromptedResult,
TrackFacesResult,
TranscribeResult,
} from "@rakazo/adapter-kit";
import { UnsupportedMediaOperationError } from "@rakazo/adapter-kit";
import type { MediaCapabilities, MediaInferenceOperation } from "@rakazo/contracts";
import {
BoxKeyframeSchema,
ContentCategorySchema,
MediaCapabilitiesSchema,
NormalizedBoxSchema,
NormalizedPointSchema,
} from "@rakazo/contracts";
import { z } from "zod";
/*
* HTTP client for the local media sidecar (infra/media-worker, API v1).
* Bearer auth, JSON bodies, per-operation timeouts, bounded retry on 503 busy,
* 501 → UnsupportedMediaOperationError, every other failure → MediaSidecarError.
* Responses are validated before they reach the caller. The token never
* appears in an error message.
*/
const DEFAULT_TIMEOUT_MS = 30 * 60_000;
const LONG_TIMEOUT_MS = 4 * 60 * 60_000;
const MAX_BUSY_RETRIES = 3;
export type MediaSidecarErrorCode =
| "bad_request"
| "invalid_path"
| "unauthorized"
| "too_large"
| "busy"
| "internal"
| "timeout"
| "unreachable"
| "malformed_response"
| "http_error";
export class MediaSidecarError extends Error {
constructor(
readonly code: MediaSidecarErrorCode,
message: string,
readonly status: number | null,
readonly operation: MediaInferenceOperation | "capabilities",
) {
super(message);
this.name = "MediaSidecarError";
}
}
// ---------------------------------------------------------------------------
// Response schemas (shapes mirror the adapter-kit result types)
// ---------------------------------------------------------------------------
const Ms = z.number().int().min(0);
const Unit = z.number().min(0).max(1);
const Span = z.object({ startMs: Ms, endMs: Ms });
const ScoredKeyframe = BoxKeyframeSchema.extend({ confidence: Unit });
const schemas = {
detectScenes: z.object({
shots: z.array(Span.extend({ score: z.number() })),
scenes: z.array(Span),
}) satisfies z.ZodType<DetectScenesResult>,
transcribe: z.object({
language: z.string().nullable(),
words: z.array(Span.extend({ text: z.string(), confidence: Unit })),
}) satisfies z.ZodType<TranscribeResult>,
detectFaces: z.object({
frames: z.array(
z.object({
tMs: Ms,
faces: z.array(
z.object({
box: NormalizedBoxSchema,
confidence: Unit,
landmarks: z.array(NormalizedPointSchema).optional(),
}),
),
}),
),
}) satisfies z.ZodType<DetectFacesResult>,
trackFaces: z.object({
sampleFps: z.number().positive(),
tracks: z.array(
z.object({
trackId: z.string().min(1),
keyframes: z.array(ScoredKeyframe),
embedding: z.array(z.number()).nullable(),
}),
),
}) satisfies z.ZodType<TrackFacesResult>,
faceLandmarks: z.object({
frames: z.array(
z.object({
tMs: Ms,
faces: z.array(
z.object({ box: NormalizedBoxSchema, landmarks: z.array(NormalizedPointSchema) }),
),
}),
),
}) satisfies z.ZodType<FaceLandmarksResult>,
embedFrames: z
.object({
model: z.string(),
dim: z.number().int().positive(),
vectors: z.array(z.array(z.number())),
})
.refine((value) => value.vectors.every((vector) => vector.length === value.dim), {
message: "vector length does not match dim",
}) satisfies z.ZodType<EmbedFramesResult>,
classifyContent: z.object({
model: z.string(),
frames: z.array(z.object({ tMs: Ms, scores: z.partialRecord(ContentCategorySchema, Unit) })),
}) satisfies z.ZodType<ClassifyContentResult>,
segmentPrompted: z.object({
maskDir: z.string().min(1),
keyframes: z.array(ScoredKeyframe),
score: Unit,
}) satisfies z.ZodType<SegmentPromptedResult>,
inpaintVideo: z.object({
outPath: z.string().min(1),
quality: z.object({ temporalStability: Unit, edgeQuality: Unit, score: Unit }),
}) satisfies z.ZodType<InpaintVideoResult>,
redact: z.object({
outPath: z.string().min(1),
framesProcessed: z.number().int().min(0),
}) satisfies z.ZodType<{ outPath: string; framesProcessed: number }>,
synthesizeIdentity: z.object({ outPath: z.string().min(1) }) satisfies z.ZodType<{
outPath: string;
}>,
audioFeatures: z.object({
integratedLufs: z.number().nullable(),
truePeakDb: z.number().nullable(),
tempoBpm: z.number().nullable(),
beatsMs: z.array(Ms),
energy: z.array(z.object({ tMs: Ms, value: z.number() })),
silences: z.array(Span),
}) satisfies z.ZodType<AudioFeaturesResult>,
saliency: z.object({
points: z.array(z.object({ tMs: Ms, cx: Unit, cy: Unit, strength: z.number() })),
}) satisfies z.ZodType<SaliencyResult>,
} as const satisfies Record<MediaInferenceOperation, z.ZodType>;
export const MEDIA_SIDECAR_RESPONSE_SCHEMAS = schemas;
const ErrorBodySchema = z.object({
error: z.object({ code: z.string(), message: z.string().optional() }),
});
// ---------------------------------------------------------------------------
// Client
// ---------------------------------------------------------------------------
export interface SidecarMediaInferenceOptions {
baseUrl: string;
token: string;
fetch?: typeof fetch;
/** Per-operation timeout overrides in ms. */
timeoutsMs?: Partial<Record<MediaInferenceOperation | "capabilities", number>>;
/** First busy-retry delay; doubles per attempt. */
retryBaseMs?: number;
}
type Request<K extends MediaInferenceOperation> = Parameters<MediaInferenceProvider[K]>[0];
type Result<K extends MediaInferenceOperation> = Awaited<ReturnType<MediaInferenceProvider[K]>>;
const KNOWN_CODES = new Set<string>([
"bad_request",
"invalid_path",
"unauthorized",
"too_large",
"busy",
"internal",
]);
export class SidecarMediaInferenceProvider implements MediaInferenceProvider {
readonly id = "sidecar";
private readonly baseUrl: string;
private readonly fetchImpl: typeof fetch;
constructor(private readonly options: SidecarMediaInferenceOptions) {
this.baseUrl = options.baseUrl.replace(/\/+$/, "");
this.fetchImpl = options.fetch ?? fetch;
}
async capabilities(signal?: AbortSignal): Promise<MediaCapabilities> {
return this.call("capabilities", "GET", "/v1/capabilities", undefined, signal, (body) =>
MediaCapabilitiesSchema.parse(body),
);
}
detectScenes(request: Request<"detectScenes">) {
return this.operation("detectScenes", request);
}
transcribe(request: Request<"transcribe">) {
return this.operation("transcribe", request);
}
detectFaces(request: Request<"detectFaces">) {
return this.operation("detectFaces", request);
}
trackFaces(request: Request<"trackFaces">) {
return this.operation("trackFaces", request);
}
faceLandmarks(request: Request<"faceLandmarks">) {
return this.operation("faceLandmarks", request);
}
embedFrames(request: Request<"embedFrames">) {
return this.operation("embedFrames", request);
}
classifyContent(request: Request<"classifyContent">) {
return this.operation("classifyContent", request);
}
segmentPrompted(request: Request<"segmentPrompted">) {
return this.operation("segmentPrompted", request);
}
inpaintVideo(request: Request<"inpaintVideo">) {
return this.operation("inpaintVideo", request);
}
redact(request: Request<"redact">) {
return this.operation("redact", request);
}
synthesizeIdentity(request: Request<"synthesizeIdentity">) {
return this.operation("synthesizeIdentity", request);
}
audioFeatures(request: Request<"audioFeatures">) {
return this.operation("audioFeatures", request);
}
saliency(request: Request<"saliency">) {
return this.operation("saliency", request);
}
private operation<K extends MediaInferenceOperation>(
operation: K,
request: Request<K>,
): Promise<Result<K>> {
const { signal, ...body } = request;
const schema = schemas[operation] as z.ZodType;
return this.call(operation, "POST", `/v1/${operation}`, body, signal, (json) => {
return schema.parse(json) as Result<K>;
});
}
private timeoutFor(operation: MediaInferenceOperation | "capabilities"): number {
const override = this.options.timeoutsMs?.[operation];
if (override !== undefined) return override;
return operation === "inpaintVideo" || operation === "synthesizeIdentity"
? LONG_TIMEOUT_MS
: DEFAULT_TIMEOUT_MS;
}
private async call<T>(
operation: MediaInferenceOperation | "capabilities",
method: "GET" | "POST",
route: string,
body: unknown,
signal: AbortSignal | undefined,
parse: (json: unknown) => T,
): Promise<T> {
signal?.throwIfAborted();
const timeout = AbortSignal.timeout(this.timeoutFor(operation));
const combined = signal ? AbortSignal.any([signal, timeout]) : timeout;
const retryBase = this.options.retryBaseMs ?? 500;
for (let attempt = 0; ; attempt++) {
let response: Response;
try {
response = await this.fetchImpl(`${this.baseUrl}${route}`, {
method,
headers: {
authorization: `Bearer ${this.options.token}`,
accept: "application/json",
...(body === undefined ? {} : { "content-type": "application/json" }),
},
body: body === undefined ? undefined : JSON.stringify(body),
signal: combined,
});
} catch (error) {
throw this.transportError(error, operation, signal, timeout);
}
if (response.status === 503 && attempt < MAX_BUSY_RETRIES) {
await response.body?.cancel().catch(() => undefined);
try {
await delay(retryBase * 2 ** attempt, combined);
} catch (error) {
throw this.transportError(error, operation, signal, timeout);
}
continue;
}
let json: unknown;
let text: string;
try {
text = await response.text();
} catch (error) {
throw this.transportError(error, operation, signal, timeout);
}
try {
json = text ? JSON.parse(text) : null;
} catch {
json = undefined;
}
if (!response.ok) throw this.httpError(response.status, json, operation);
if (json === undefined)
throw new MediaSidecarError(
"malformed_response",
`media sidecar returned invalid JSON for ${operation}`,
response.status,
operation,
);
try {
return parse(json);
} catch (error) {
const detail =
error instanceof z.ZodError
? error.issues
.slice(0, 3)
.map((issue) => `${issue.path.join(".") || "(root)"}: ${issue.message}`)
.join("; ")
: "unexpected shape";
throw new MediaSidecarError(
"malformed_response",
this.scrub(`media sidecar returned a malformed ${operation} response: ${detail}`),
response.status,
operation,
);
}
}
}
private httpError(
status: number,
json: unknown,
operation: MediaInferenceOperation | "capabilities",
): Error {
const parsed = ErrorBodySchema.safeParse(json);
const message = this.scrub(
parsed.success && parsed.data.error.message
? parsed.data.error.message
: `media sidecar responded ${status}`,
);
if (status === 501 && operation !== "capabilities")
return new UnsupportedMediaOperationError(operation, message);
const serverCode = parsed.success ? parsed.data.error.code : "";
const code: MediaSidecarErrorCode = KNOWN_CODES.has(serverCode)
? (serverCode as MediaSidecarErrorCode)
: status === 401
? "unauthorized"
: status === 503
? "busy"
: "http_error";
return new MediaSidecarError(code, `${operation}: ${message}`, status, operation);
}
private transportError(
error: unknown,
operation: MediaInferenceOperation | "capabilities",
signal: AbortSignal | undefined,
timeout: AbortSignal,
): unknown {
if (signal?.aborted) return signal.reason;
if (timeout.aborted)
return new MediaSidecarError(
"timeout",
`media sidecar ${operation} timed out`,
null,
operation,
);
const detail = error instanceof Error ? error.message : String(error);
return new MediaSidecarError(
"unreachable",
this.scrub(`media sidecar is unreachable: ${detail}`),
null,
operation,
);
}
/** Strip the bearer token from any text that may end up in logs or errors. */
private scrub(text: string): string {
const token = this.options.token;
return token ? text.split(token).join("[redacted]") : text;
}
}
function delay(ms: number, signal: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
if (signal.aborted) {
reject(signal.reason);
return;
}
const timer = setTimeout(() => {
signal.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
reject(signal.reason);
};
signal.addEventListener("abort", onAbort, { once: true });
});
}

View File

@ -0,0 +1,60 @@
import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { listProjectWorkspaceFiles, readProjectWorkspaceFile } from "./project-workspace-files.js";
describe("project workspace files", () => {
const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) {
rmSync(root, { recursive: true, force: true });
}
});
function linkedFolderBesideSecret() {
const base = mkdtempSync(path.join(tmpdir(), "project-workspace-files-"));
roots.push(base);
const project = path.join(base, "project");
const outside = path.join(base, "outside");
mkdirSync(path.join(project, "docs"), { recursive: true });
mkdirSync(outside);
writeFileSync(path.join(project, "docs", "readme.md"), "inside");
writeFileSync(path.join(outside, "secret.txt"), "outside");
return { project, outside };
}
it("lists and reads files inside the linked folder, including symlinks that stay inside", async () => {
const { project } = linkedFolderBesideSecret();
symlinkSync(path.join(project, "docs", "readme.md"), path.join(project, "readme-link.md"));
const listed = await listProjectWorkspaceFiles(project, "project");
expect(listed.map((entry) => entry.path).sort()).toEqual([
"project/docs",
"project/readme-link.md",
]);
expect(await listProjectWorkspaceFiles(project, "project/docs")).toEqual([
{ path: "project/docs/readme.md", kind: "file", size: 6 },
]);
const read = await readProjectWorkspaceFile(project, "project/readme-link.md");
expect(new TextDecoder().decode(read)).toBe("inside");
});
it("keeps symlinks from leaving the linked folder", async () => {
const { project, outside } = linkedFolderBesideSecret();
symlinkSync(outside, path.join(project, "escape"));
symlinkSync(path.join(outside, "secret.txt"), path.join(project, "secret-link.txt"));
symlinkSync(path.join(project, "missing"), path.join(project, "dangling"));
const listed = await listProjectWorkspaceFiles(project, "project");
expect(listed.map((entry) => entry.path)).toEqual(["project/docs"]);
await expect(listProjectWorkspaceFiles(project, "project/escape")).rejects.toThrow(/escapes/i);
await expect(readProjectWorkspaceFile(project, "project/escape/secret.txt")).rejects.toThrow(
/escapes/i,
);
await expect(readProjectWorkspaceFile(project, "project/secret-link.txt")).rejects.toThrow(
/escapes/i,
);
});
});

View File

@ -1,6 +1,8 @@
import { readdir, readFile, stat } from "node:fs/promises";
import { readdir, readFile, realpath, stat } from "node:fs/promises";
import path from "node:path";
import type { ComputerFileEntry } from "@rakazo/adapter-kit";
import { normalizeWorkspacePath } from "./computer-support.js";
import { isAllowedDesktopPath } from "./desktop-sandbox-paths.js";
import {
isProjectWorkspacePath,
projectWorkspaceRelativePath,
@ -14,28 +16,34 @@ export async function listProjectWorkspaceFiles(
if (!isProjectWorkspacePath(storedPath)) {
throw new Error("Path is not under the project workspace");
}
const hostDirectory = resolveProjectHostPath(workspaceRoot, storedPath);
const root = await realpath(workspaceRoot);
const hostDirectory = await containedHostPath(
root,
resolveProjectHostPath(workspaceRoot, storedPath),
);
const relativeDirectory = projectWorkspaceRelativePath(storedPath);
const entries = await readdir(hostDirectory, { withFileTypes: true });
const listed = await Promise.all(
entries.map(async (entry) => {
const childHost = resolveProjectHostPath(
workspaceRoot,
relativeDirectory ? `project/${relativeDirectory}/${entry.name}` : `project/${entry.name}`,
);
const info = await stat(childHost);
// A symlink that leaves the linked folder, or leads nowhere, is not part of the project.
const info = await containedHostPath(root, path.join(hostDirectory, entry.name))
.then((target) => stat(target))
.catch(() => null);
if (!info) return [];
const childStored = normalizeWorkspacePath(
relativeDirectory ? `project/${relativeDirectory}/${entry.name}` : `project/${entry.name}`,
);
return {
path: childStored,
kind: info.isDirectory() ? ("dir" as const) : ("file" as const),
size: info.size,
...(info.isFile() && info.mode & 0o100 ? { executable: true } : {}),
};
return [
{
path: childStored,
kind: info.isDirectory() ? ("dir" as const) : ("file" as const),
size: info.size,
...(info.isFile() && info.mode & 0o100 ? { executable: true } : {}),
},
];
}),
);
return listed;
return listed.flat();
}
export async function readProjectWorkspaceFile(
@ -43,10 +51,22 @@ export async function readProjectWorkspaceFile(
storedPath: string,
options?: { maxBytes?: number },
): Promise<Uint8Array> {
const hostPath = resolveProjectHostPath(workspaceRoot, storedPath);
const hostPath = await containedHostPath(
await realpath(workspaceRoot),
resolveProjectHostPath(workspaceRoot, storedPath),
);
const info = await stat(hostPath);
if (options?.maxBytes !== undefined && info.size > options.maxBytes) {
throw new Error(`computer file exceeds ${options.maxBytes} bytes`);
}
return new Uint8Array(await readFile(hostPath));
}
/** Follows symlinks, but only to places inside the linked folder. */
async function containedHostPath(root: string, candidate: string): Promise<string> {
const target = await realpath(candidate);
if (!isAllowedDesktopPath(target, [root])) {
throw new Error("Path escapes the project workspace");
}
return target;
}

View File

@ -24,6 +24,12 @@ function handlers(): BackgroundJobHandlers {
"messaging.deliver": vi.fn(async () => undefined),
"cloud_agent.poll": vi.fn(async () => undefined),
"profile.consolidate": vi.fn(async () => undefined),
"media.sync": vi.fn(async () => undefined),
"media.proxy": vi.fn(async () => undefined),
"media.analyze": vi.fn(async () => undefined),
"media.cluster": vi.fn(async () => undefined),
"media.render": vi.fn(async () => undefined),
"media.publish": vi.fn(async () => undefined),
};
}

View File

@ -29,6 +29,12 @@ function handlers(overrides: Partial<BackgroundJobHandlers> = {}): BackgroundJob
"messaging.deliver": vi.fn(async () => undefined),
"cloud_agent.poll": vi.fn(async () => undefined),
"profile.consolidate": vi.fn(async () => undefined),
"media.sync": vi.fn(async () => undefined),
"media.proxy": vi.fn(async () => undefined),
"media.analyze": vi.fn(async () => undefined),
"media.cluster": vi.fn(async () => undefined),
"media.render": vi.fn(async () => undefined),
"media.publish": vi.fn(async () => undefined),
...overrides,
};
}

View File

@ -14,6 +14,12 @@ function handlers(): BackgroundJobHandlers {
"messaging.deliver": vi.fn(async () => undefined),
"cloud_agent.poll": vi.fn(async () => undefined),
"profile.consolidate": vi.fn(async () => undefined),
"media.sync": vi.fn(async () => undefined),
"media.proxy": vi.fn(async () => undefined),
"media.analyze": vi.fn(async () => undefined),
"media.cluster": vi.fn(async () => undefined),
"media.render": vi.fn(async () => undefined),
"media.publish": vi.fn(async () => undefined),
};
}

View File

@ -9,6 +9,7 @@ export * from "./inbox.js";
export * from "./integration-settings.js";
export * from "./local-settings.js";
export * from "./mcp.js";
export * from "./media.js";
export * from "./model-tiers.js";
export * from "./openai-compatible-ui.js";
export * from "./project-goals.js";

File diff suppressed because it is too large Load Diff

View File

@ -82,6 +82,7 @@ import {
IntegrationProviderConfigSchema,
IntegrationSetupStateSchema,
} from "./integration-settings.js";
import { mediaRpcShapes } from "./media.js";
import { ModelTierMappingSchema, SetModelTierInput } from "./model-tiers.js";
import {
CreateProjectGoalInput,
@ -157,6 +158,10 @@ const threadSendInput = threadTarget
}
});
function rpc<I extends z.ZodType, O extends z.ZodType>(shape: { input: I; output: O }) {
return oc.input(shape.input).output(shape.output);
}
export const appContract = {
aiConsent: {
status: oc.input(AiConsentQuerySchema).output(AiConsentStatusSchema),
@ -861,6 +866,96 @@ export const appContract = {
put: oc.input(AgentSecretInputSchema).output(AgentSecretSchema),
remove: oc.input(z.object({ id: Id })).output(z.object({ ok: z.literal(true) })),
},
media: {
capabilities: rpc(mediaRpcShapes.capabilities),
sources: {
list: rpc(mediaRpcShapes.sources.list),
createFolder: rpc(mediaRpcShapes.sources.createFolder),
registerClient: rpc(mediaRpcShapes.sources.registerClient),
pushItems: rpc(mediaRpcShapes.sources.pushItems),
sync: rpc(mediaRpcShapes.sources.sync),
remove: rpc(mediaRpcShapes.sources.remove),
},
assets: {
search: rpc(mediaRpcShapes.assets.search),
get: rpc(mediaRpcShapes.assets.get),
setPaidOnly: rpc(mediaRpcShapes.assets.setPaidOnly),
requestUpload: rpc(mediaRpcShapes.assets.requestUpload),
},
uploads: {
begin: rpc(mediaRpcShapes.uploads.begin),
status: rpc(mediaRpcShapes.uploads.status),
complete: rpc(mediaRpcShapes.uploads.complete),
},
collections: {
list: rpc(mediaRpcShapes.collections.list),
get: rpc(mediaRpcShapes.collections.get),
rename: rpc(mediaRpcShapes.collections.rename),
merge: rpc(mediaRpcShapes.collections.merge),
split: rpc(mediaRpcShapes.collections.split),
createManual: rpc(mediaRpcShapes.collections.createManual),
recluster: rpc(mediaRpcShapes.collections.recluster),
},
people: {
list: rpc(mediaRpcShapes.people.list),
update: rpc(mediaRpcShapes.people.update),
merge: rpc(mediaRpcShapes.people.merge),
assignTrack: rpc(mediaRpcShapes.people.assignTrack),
recordConsent: rpc(mediaRpcShapes.people.recordConsent),
remove: rpc(mediaRpcShapes.people.remove),
},
marks: {
list: rpc(mediaRpcShapes.marks.list),
create: rpc(mediaRpcShapes.marks.create),
update: rpc(mediaRpcShapes.marks.update),
remove: rpc(mediaRpcShapes.marks.remove),
},
brandKits: {
list: rpc(mediaRpcShapes.brandKits.list),
upsert: rpc(mediaRpcShapes.brandKits.upsert),
remove: rpc(mediaRpcShapes.brandKits.remove),
templates: rpc(mediaRpcShapes.brandKits.templates),
introVariants: rpc(mediaRpcShapes.brandKits.introVariants),
},
profiles: {
list: rpc(mediaRpcShapes.profiles.list),
get: rpc(mediaRpcShapes.profiles.get),
},
projects: {
list: rpc(mediaRpcShapes.projects.list),
get: rpc(mediaRpcShapes.projects.get),
create: rpc(mediaRpcShapes.projects.create),
update: rpc(mediaRpcShapes.projects.update),
remove: rpc(mediaRpcShapes.projects.remove),
autoPlan: rpc(mediaRpcShapes.projects.autoPlan),
},
plans: {
submit: rpc(mediaRpcShapes.plans.submit),
},
renders: {
start: rpc(mediaRpcShapes.renders.start),
get: rpc(mediaRpcShapes.renders.get),
cancel: rpc(mediaRpcShapes.renders.cancel),
thumbnails: rpc(mediaRpcShapes.renders.thumbnails),
},
marketing: {
copy: rpc(mediaRpcShapes.marketing.copy),
},
publications: {
list: rpc(mediaRpcShapes.publications.list),
create: rpc(mediaRpcShapes.publications.create),
bundle: rpc(mediaRpcShapes.publications.bundle),
markPublished: rpc(mediaRpcShapes.publications.markPublished),
cancel: rpc(mediaRpcShapes.publications.cancel),
},
metrics: {
mappings: rpc(mediaRpcShapes.metrics.mappings),
saveMapping: rpc(mediaRpcShapes.metrics.saveMapping),
importCsv: rpc(mediaRpcShapes.metrics.importCsv),
record: rpc(mediaRpcShapes.metrics.record),
funnel: rpc(mediaRpcShapes.metrics.funnel),
},
},
};
export type AppContract = typeof appContract;

View File

@ -1,19 +1,10 @@
// Server-only. Kept out of the package index so browser bundles never load `node:child_process`.
import { spawn } from "node:child_process";
import { access } from "node:fs/promises";
import path from "node:path";
import { gitStatusArgv, gitUntrackedFilesArgv, parseGitNameOnly } from "./compose-update.js";
import { DEFAULT_UPDATE_REMOTE, parseGitStatusPorcelain } from "./self-update.js";
// Lazy import to avoid bundling `node:child_process` in browser bundles.
// `defaultGitRunner` is server-only; lazy-loading ensures tree-shaking works.
type SpawnFn = typeof import("node:child_process").spawn;
let _spawn: SpawnFn | undefined;
const getSpawn = (): SpawnFn => {
if (_spawn === undefined) {
_spawn = require("node:child_process").spawn as SpawnFn;
}
return _spawn;
};
export interface GitCheckoutStatus {
present: boolean;
remoteUrl: string | null;
@ -32,7 +23,6 @@ export type GitRunner = (args: string[], cwd: string) => Promise<GitRunnerResult
export async function defaultGitRunner(args: string[], cwd: string): Promise<GitRunnerResult> {
return new Promise((resolve) => {
const spawn = getSpawn();
const child = spawn("git", args, { cwd, env: process.env });
let output = "";
child.stdout.on("data", (chunk: Buffer | string) => {

View File

@ -18,14 +18,15 @@
"test": "vitest run --root ../.. packages/db/src"
},
"dependencies": {
"@rakazo/adapter-kit": "workspace:*",
"@prisma/adapter-pg": "7.10.0",
"@prisma/client": "7.10.0",
"@rakazo/adapter-kit": "workspace:*",
"@rakazo/contracts": "workspace:*",
"@rakazo/core": "workspace:*",
"@rakazo/logging": "workspace:*",
"pg": "8.23.0",
"dotenv": "17.4.2"
"@rakazo/media": "workspace:*",
"dotenv": "17.4.2",
"pg": "8.23.0"
},
"devDependencies": {
"@types/pg": "8.23.1",

View File

@ -0,0 +1,513 @@
-- CreateTable
CREATE TABLE "media_sources" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"ownerUserId" TEXT NOT NULL,
"kind" TEXT NOT NULL,
"label" TEXT NOT NULL,
"config" JSONB NOT NULL DEFAULT '{}',
"status" TEXT NOT NULL DEFAULT 'idle',
"error" TEXT,
"lastSyncCursor" TEXT,
"lastSyncedAt" TIMESTAMP(3),
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_sources_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_assets" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"sourceId" TEXT NOT NULL,
"externalId" TEXT NOT NULL,
"revision" TEXT,
"contentHash" TEXT,
"kind" TEXT NOT NULL,
"storage" TEXT NOT NULL,
"storagePath" TEXT,
"fileName" TEXT,
"sizeBytes" BIGINT,
"capturedAt" TIMESTAMP(3),
"durationMs" INTEGER,
"width" INTEGER,
"height" INTEGER,
"fps" DOUBLE PRECISION,
"codec" TEXT,
"orientation" INTEGER,
"gpsLat" DOUBLE PRECISION,
"gpsLon" DOUBLE PRECISION,
"deviceModel" TEXT,
"albumNames" TEXT[] DEFAULT ARRAY[]::TEXT[],
"isFavorite" BOOLEAN NOT NULL DEFAULT false,
"proxyPath" TEXT,
"thumbPath" TEXT,
"spritePath" TEXT,
"qualityScore" DOUBLE PRECISION,
"analysisVersion" INTEGER NOT NULL DEFAULT 0,
"ingestState" TEXT NOT NULL DEFAULT 'indexed',
"ingestError" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_assets_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_uploads" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"assetId" TEXT NOT NULL,
"userId" TEXT NOT NULL,
"sizeBytes" BIGINT NOT NULL,
"chunkBytes" INTEGER NOT NULL,
"sha256" TEXT NOT NULL,
"fileName" TEXT NOT NULL,
"receivedChunks" INTEGER[] DEFAULT ARRAY[]::INTEGER[],
"status" TEXT NOT NULL DEFAULT 'open',
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_uploads_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_segments" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"assetId" TEXT NOT NULL,
"kind" TEXT NOT NULL,
"startMs" INTEGER NOT NULL,
"endMs" INTEGER NOT NULL,
"scores" JSONB NOT NULL DEFAULT '{}',
"teaserScore" DOUBLE PRECISION,
"transcriptExcerpt" TEXT,
"labels" TEXT[] DEFAULT ARRAY[]::TEXT[],
"paidOnly" BOOLEAN NOT NULL DEFAULT false,
"analysisVersion" INTEGER NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "media_segments_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_analyses" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"assetId" TEXT NOT NULL,
"kind" TEXT NOT NULL,
"modelId" TEXT NOT NULL,
"modelVersion" TEXT NOT NULL,
"data" JSONB,
"dataPath" TEXT,
"vectors" BYTEA,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "media_analyses_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_collections" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"kind" TEXT NOT NULL,
"title" TEXT NOT NULL,
"labelSource" TEXT NOT NULL DEFAULT 'rule',
"startAt" TIMESTAMP(3),
"endAt" TIMESTAMP(3),
"centroidLat" DOUBLE PRECISION,
"centroidLon" DOUBLE PRECISION,
"coverAssetId" TEXT,
"clusterKey" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_collections_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_collection_items" (
"collectionId" TEXT NOT NULL,
"assetId" TEXT NOT NULL,
"position" INTEGER NOT NULL,
CONSTRAINT "media_collection_items_pkey" PRIMARY KEY ("collectionId","assetId")
);
-- CreateTable
CREATE TABLE "media_people" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"displayName" TEXT NOT NULL,
"isSelf" BOOLEAN NOT NULL DEFAULT false,
"confirmed" BOOLEAN NOT NULL DEFAULT false,
"anonymizeByDefault" TEXT NOT NULL DEFAULT 'none',
"centroid" BYTEA,
"consentArtifactId" TEXT,
"consentRecordedAt" TIMESTAMP(3),
"consentRecordedBy" TEXT,
"ageVerifiedAt" TIMESTAMP(3),
"ageVerificationMethod" TEXT NOT NULL DEFAULT 'none',
"syntheticSeed" INTEGER,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_people_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_face_tracks" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"assetId" TEXT NOT NULL,
"personId" TEXT,
"startMs" INTEGER NOT NULL,
"endMs" INTEGER NOT NULL,
"confidenceMin" DOUBLE PRECISION NOT NULL,
"keyframesPath" TEXT NOT NULL,
"embedding" BYTEA,
"modelVersion" TEXT NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "media_face_tracks_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_marks" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"kind" TEXT NOT NULL,
"label" TEXT,
"personId" TEXT,
"assetId" TEXT,
"prompt" JSONB,
"treatment" TEXT NOT NULL,
"maskDir" TEXT,
"qualityScore" DOUBLE PRECISION,
"suggestedFallback" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_marks_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_brand_kits" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"kit" JSONB NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_brand_kits_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_projects" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"createdBy" TEXT NOT NULL,
"title" TEXT NOT NULL,
"goal" TEXT NOT NULL,
"status" TEXT NOT NULL DEFAULT 'draft',
"brandKitId" TEXT,
"collectionIds" TEXT[] DEFAULT ARRAY[]::TEXT[],
"assetIds" TEXT[] DEFAULT ARRAY[]::TEXT[],
"profileIds" TEXT[] DEFAULT ARRAY[]::TEXT[],
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_projects_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_edit_plans" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"projectId" TEXT NOT NULL,
"profileId" TEXT NOT NULL,
"version" INTEGER NOT NULL,
"edl" JSONB NOT NULL,
"createdBy" TEXT NOT NULL,
"rationale" TEXT,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "media_edit_plans_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_renders" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"projectId" TEXT NOT NULL,
"planId" TEXT NOT NULL,
"profileId" TEXT NOT NULL,
"quality" TEXT NOT NULL,
"status" TEXT NOT NULL DEFAULT 'queued',
"progress" DOUBLE PRECISION NOT NULL DEFAULT 0,
"outputs" JSONB NOT NULL DEFAULT '[]',
"gateReport" JSONB,
"ffprobe" JSONB,
"autoFixRound" INTEGER NOT NULL DEFAULT 0,
"error" TEXT,
"createdBy" TEXT NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_renders_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_publications" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"renderId" TEXT NOT NULL,
"profileId" TEXT NOT NULL,
"destination" TEXT NOT NULL,
"status" TEXT NOT NULL DEFAULT 'draft',
"scheduledAt" TIMESTAMP(3),
"publishedAt" TIMESTAMP(3),
"externalUrl" TEXT,
"targetUrl" TEXT,
"trackingUrl" TEXT,
"utm" JSONB,
"title" TEXT,
"caption" TEXT,
"tags" TEXT[] DEFAULT ARRAY[]::TEXT[],
"thumbnailVariant" TEXT,
"bundleArtifactId" TEXT,
"createdBy" TEXT NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_publications_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_metrics" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"publicationId" TEXT NOT NULL,
"observedAt" TIMESTAMP(3) NOT NULL,
"views" INTEGER NOT NULL DEFAULT 0,
"clicks" INTEGER NOT NULL DEFAULT 0,
"conversions" INTEGER NOT NULL DEFAULT 0,
"revenueCents" INTEGER NOT NULL DEFAULT 0,
"currency" TEXT NOT NULL DEFAULT 'EUR',
"source" TEXT NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "media_metrics_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_metric_mappings" (
"id" TEXT NOT NULL,
"spaceId" TEXT NOT NULL,
"name" TEXT NOT NULL,
"mapping" JSONB NOT NULL,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_metric_mappings_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "media_teaser_models" (
"spaceId" TEXT NOT NULL,
"weights" JSONB NOT NULL,
"sampleCount" INTEGER NOT NULL,
"updatedAt" TIMESTAMP(3) NOT NULL,
CONSTRAINT "media_teaser_models_pkey" PRIMARY KEY ("spaceId")
);
-- CreateIndex
CREATE INDEX "media_sources_spaceId_createdAt_idx" ON "media_sources"("spaceId", "createdAt");
-- CreateIndex
CREATE INDEX "media_assets_spaceId_capturedAt_idx" ON "media_assets"("spaceId", "capturedAt");
-- CreateIndex
CREATE INDEX "media_assets_spaceId_kind_capturedAt_idx" ON "media_assets"("spaceId", "kind", "capturedAt");
-- CreateIndex
CREATE INDEX "media_assets_spaceId_ingestState_idx" ON "media_assets"("spaceId", "ingestState");
-- CreateIndex
CREATE INDEX "media_assets_contentHash_idx" ON "media_assets"("contentHash");
-- CreateIndex
CREATE UNIQUE INDEX "media_assets_sourceId_externalId_key" ON "media_assets"("sourceId", "externalId");
-- CreateIndex
CREATE INDEX "media_uploads_assetId_idx" ON "media_uploads"("assetId");
-- CreateIndex
CREATE INDEX "media_uploads_spaceId_status_idx" ON "media_uploads"("spaceId", "status");
-- CreateIndex
CREATE INDEX "media_segments_assetId_startMs_idx" ON "media_segments"("assetId", "startMs");
-- CreateIndex
CREATE INDEX "media_segments_spaceId_teaserScore_idx" ON "media_segments"("spaceId", "teaserScore");
-- CreateIndex
CREATE INDEX "media_analyses_spaceId_kind_idx" ON "media_analyses"("spaceId", "kind");
-- CreateIndex
CREATE UNIQUE INDEX "media_analyses_assetId_kind_modelVersion_key" ON "media_analyses"("assetId", "kind", "modelVersion");
-- CreateIndex
CREATE INDEX "media_collections_spaceId_kind_startAt_idx" ON "media_collections"("spaceId", "kind", "startAt");
-- CreateIndex
CREATE UNIQUE INDEX "media_collections_spaceId_kind_clusterKey_key" ON "media_collections"("spaceId", "kind", "clusterKey");
-- CreateIndex
CREATE INDEX "media_collection_items_assetId_idx" ON "media_collection_items"("assetId");
-- CreateIndex
CREATE INDEX "media_people_spaceId_idx" ON "media_people"("spaceId");
-- CreateIndex
CREATE INDEX "media_face_tracks_assetId_idx" ON "media_face_tracks"("assetId");
-- CreateIndex
CREATE INDEX "media_face_tracks_personId_idx" ON "media_face_tracks"("personId");
-- CreateIndex
CREATE INDEX "media_marks_spaceId_personId_idx" ON "media_marks"("spaceId", "personId");
-- CreateIndex
CREATE INDEX "media_marks_assetId_idx" ON "media_marks"("assetId");
-- CreateIndex
CREATE INDEX "media_brand_kits_spaceId_idx" ON "media_brand_kits"("spaceId");
-- CreateIndex
CREATE INDEX "media_projects_spaceId_updatedAt_idx" ON "media_projects"("spaceId", "updatedAt");
-- CreateIndex
CREATE UNIQUE INDEX "media_edit_plans_projectId_profileId_version_key" ON "media_edit_plans"("projectId", "profileId", "version");
-- CreateIndex
CREATE INDEX "media_renders_projectId_createdAt_idx" ON "media_renders"("projectId", "createdAt");
-- CreateIndex
CREATE INDEX "media_renders_spaceId_status_idx" ON "media_renders"("spaceId", "status");
-- CreateIndex
CREATE INDEX "media_publications_spaceId_createdAt_idx" ON "media_publications"("spaceId", "createdAt");
-- CreateIndex
CREATE INDEX "media_publications_renderId_idx" ON "media_publications"("renderId");
-- CreateIndex
CREATE INDEX "media_metrics_spaceId_observedAt_idx" ON "media_metrics"("spaceId", "observedAt");
-- CreateIndex
CREATE UNIQUE INDEX "media_metrics_publicationId_observedAt_source_key" ON "media_metrics"("publicationId", "observedAt", "source");
-- CreateIndex
CREATE UNIQUE INDEX "media_metric_mappings_spaceId_name_key" ON "media_metric_mappings"("spaceId", "name");
-- AddForeignKey
ALTER TABLE "media_sources" ADD CONSTRAINT "media_sources_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_assets" ADD CONSTRAINT "media_assets_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_assets" ADD CONSTRAINT "media_assets_sourceId_fkey" FOREIGN KEY ("sourceId") REFERENCES "media_sources"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_uploads" ADD CONSTRAINT "media_uploads_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_uploads" ADD CONSTRAINT "media_uploads_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_segments" ADD CONSTRAINT "media_segments_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_segments" ADD CONSTRAINT "media_segments_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_analyses" ADD CONSTRAINT "media_analyses_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_analyses" ADD CONSTRAINT "media_analyses_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_collections" ADD CONSTRAINT "media_collections_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_collection_items" ADD CONSTRAINT "media_collection_items_collectionId_fkey" FOREIGN KEY ("collectionId") REFERENCES "media_collections"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_collection_items" ADD CONSTRAINT "media_collection_items_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_people" ADD CONSTRAINT "media_people_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_face_tracks" ADD CONSTRAINT "media_face_tracks_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_face_tracks" ADD CONSTRAINT "media_face_tracks_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_face_tracks" ADD CONSTRAINT "media_face_tracks_personId_fkey" FOREIGN KEY ("personId") REFERENCES "media_people"("id") ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_marks" ADD CONSTRAINT "media_marks_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_marks" ADD CONSTRAINT "media_marks_personId_fkey" FOREIGN KEY ("personId") REFERENCES "media_people"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_marks" ADD CONSTRAINT "media_marks_assetId_fkey" FOREIGN KEY ("assetId") REFERENCES "media_assets"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_brand_kits" ADD CONSTRAINT "media_brand_kits_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_projects" ADD CONSTRAINT "media_projects_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_projects" ADD CONSTRAINT "media_projects_brandKitId_fkey" FOREIGN KEY ("brandKitId") REFERENCES "media_brand_kits"("id") ON DELETE SET NULL ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_edit_plans" ADD CONSTRAINT "media_edit_plans_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_edit_plans" ADD CONSTRAINT "media_edit_plans_projectId_fkey" FOREIGN KEY ("projectId") REFERENCES "media_projects"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_renders" ADD CONSTRAINT "media_renders_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_renders" ADD CONSTRAINT "media_renders_projectId_fkey" FOREIGN KEY ("projectId") REFERENCES "media_projects"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_renders" ADD CONSTRAINT "media_renders_planId_fkey" FOREIGN KEY ("planId") REFERENCES "media_edit_plans"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_publications" ADD CONSTRAINT "media_publications_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_publications" ADD CONSTRAINT "media_publications_renderId_fkey" FOREIGN KEY ("renderId") REFERENCES "media_renders"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_metrics" ADD CONSTRAINT "media_metrics_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_metrics" ADD CONSTRAINT "media_metrics_publicationId_fkey" FOREIGN KEY ("publicationId") REFERENCES "media_publications"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_metric_mappings" ADD CONSTRAINT "media_metric_mappings_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;
-- AddForeignKey
ALTER TABLE "media_teaser_models" ADD CONSTRAINT "media_teaser_models_spaceId_fkey" FOREIGN KEY ("spaceId") REFERENCES "spaces"("id") ON DELETE CASCADE ON UPDATE CASCADE;

View File

@ -216,6 +216,23 @@ model Space {
userFactVisibility UserFactSpace[]
delegatedTasks DelegatedTask[]
semanticMemoryItems SemanticMemoryItem[]
mediaSources MediaSource[]
mediaAssets MediaAsset[]
mediaUploads MediaUpload[]
mediaSegments MediaSegment[]
mediaAnalyses MediaAnalysis[]
mediaCollections MediaCollection[]
mediaPeople MediaPerson[]
mediaFaceTracks MediaFaceTrack[]
mediaMarks MediaMark[]
mediaBrandKits MediaBrandKit[]
mediaProjects MediaProject[]
mediaEditPlans MediaEditPlan[]
mediaRenders MediaRender[]
mediaPublications MediaPublication[]
mediaMetrics MediaMetric[]
mediaMetricMappings MediaMetricMapping[]
mediaTeaserModel MediaTeaserModel?
@@unique([id, organizationId])
@@index([organizationId, createdAt])
@ -1497,3 +1514,411 @@ model DelegatedTask {
// CHECK constraint delegated_tasks_status_chk is created in SQL.
@@map("delegated_tasks")
}
// ---------------------------------------------------------------------------
// Media Studio
// ---------------------------------------------------------------------------
model MediaSource {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
ownerUserId String
kind String
label String
/// Folder: { path }. Client-pushed: { deviceId }. Never secrets.
config Json @default("{}")
status String @default("idle")
error String?
lastSyncCursor String?
lastSyncedAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
assets MediaAsset[]
@@index([spaceId, createdAt])
@@map("media_sources")
}
model MediaAsset {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
sourceId String
source MediaSource @relation(fields: [sourceId], references: [id], onDelete: Cascade)
externalId String
revision String?
contentHash String?
kind String
storage String
/// Uploaded originals: path relative to the media root. Referenced originals resolve through the source.
storagePath String?
fileName String?
sizeBytes BigInt?
capturedAt DateTime?
durationMs Int?
width Int?
height Int?
fps Float?
codec String?
orientation Int?
/// Rounded to two decimals (about 1 km).
gpsLat Float?
gpsLon Float?
deviceModel String?
albumNames String[] @default([])
isFavorite Boolean @default(false)
proxyPath String?
thumbPath String?
spritePath String?
qualityScore Float?
analysisVersion Int @default(0)
ingestState String @default("indexed")
ingestError String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
segments MediaSegment[]
analyses MediaAnalysis[]
collectionItems MediaCollectionItem[]
faceTracks MediaFaceTrack[]
marks MediaMark[]
uploads MediaUpload[]
@@unique([sourceId, externalId])
@@index([spaceId, capturedAt])
@@index([spaceId, kind, capturedAt])
@@index([spaceId, ingestState])
@@index([contentHash])
@@map("media_assets")
}
model MediaUpload {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
assetId String
asset MediaAsset @relation(fields: [assetId], references: [id], onDelete: Cascade)
userId String
sizeBytes BigInt
chunkBytes Int
sha256 String
fileName String
receivedChunks Int[] @default([])
status String @default("open")
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@index([assetId])
@@index([spaceId, status])
@@map("media_uploads")
}
model MediaSegment {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
assetId String
asset MediaAsset @relation(fields: [assetId], references: [id], onDelete: Cascade)
kind String
startMs Int
endMs Int
/// Partial<Record<TeaserFeature, number>>
scores Json @default("{}")
teaserScore Float?
transcriptExcerpt String?
labels String[] @default([])
paidOnly Boolean @default(false)
analysisVersion Int
createdAt DateTime @default(now())
@@index([assetId, startMs])
@@index([spaceId, teaserScore])
@@map("media_segments")
}
model MediaAnalysis {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
assetId String
asset MediaAsset @relation(fields: [assetId], references: [id], onDelete: Cascade)
kind String
modelId String
modelVersion String
/// Small results inline; large results as a path relative to the media root.
data Json?
dataPath String?
/// Float32 little-endian vectors for embeddings.
vectors Bytes?
createdAt DateTime @default(now())
@@unique([assetId, kind, modelVersion])
@@index([spaceId, kind])
@@map("media_analyses")
}
model MediaCollection {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
kind String
title String
labelSource String @default("rule")
startAt DateTime?
endAt DateTime?
centroidLat Float?
centroidLon Float?
coverAssetId String?
/// Stable key so re-clustering updates instead of recreating user-renamed collections.
clusterKey String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
items MediaCollectionItem[]
@@unique([spaceId, kind, clusterKey])
@@index([spaceId, kind, startAt])
@@map("media_collections")
}
model MediaCollectionItem {
collectionId String
collection MediaCollection @relation(fields: [collectionId], references: [id], onDelete: Cascade)
assetId String
asset MediaAsset @relation(fields: [assetId], references: [id], onDelete: Cascade)
position Int
@@id([collectionId, assetId])
@@index([assetId])
@@map("media_collection_items")
}
model MediaPerson {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
displayName String
isSelf Boolean @default(false)
confirmed Boolean @default(false)
anonymizeByDefault String @default("none")
/// Mean identity embedding (Float32). Biometric: local only, deleted with the person.
centroid Bytes?
consentArtifactId String?
consentRecordedAt DateTime?
consentRecordedBy String?
ageVerifiedAt DateTime?
ageVerificationMethod String @default("none")
/// Seed for a stable synthetic identity.
syntheticSeed Int?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
faceTracks MediaFaceTrack[]
marks MediaMark[]
@@index([spaceId])
@@map("media_people")
}
model MediaFaceTrack {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
assetId String
asset MediaAsset @relation(fields: [assetId], references: [id], onDelete: Cascade)
personId String?
person MediaPerson? @relation(fields: [personId], references: [id], onDelete: SetNull)
startMs Int
endMs Int
confidenceMin Float
/// Box keyframes JSON relative to the media root.
keyframesPath String
/// Biometric (Float32). Local only.
embedding Bytes?
modelVersion String
createdAt DateTime @default(now())
@@index([assetId])
@@index([personId])
@@map("media_face_tracks")
}
model MediaMark {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
kind String
label String?
personId String?
person MediaPerson? @relation(fields: [personId], references: [id], onDelete: Cascade)
assetId String?
asset MediaAsset? @relation(fields: [assetId], references: [id], onDelete: Cascade)
/// MediaMarkPrompt
prompt Json?
treatment String
maskDir String?
qualityScore Float?
suggestedFallback String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@index([spaceId, personId])
@@index([assetId])
@@map("media_marks")
}
model MediaBrandKit {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
/// BrandKit
kit Json
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
projects MediaProject[]
@@index([spaceId])
@@map("media_brand_kits")
}
model MediaProject {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
createdBy String
title String
goal String
status String @default("draft")
brandKitId String?
brandKit MediaBrandKit? @relation(fields: [brandKitId], references: [id], onDelete: SetNull)
collectionIds String[] @default([])
assetIds String[] @default([])
profileIds String[] @default([])
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
plans MediaEditPlan[]
renders MediaRender[]
@@index([spaceId, updatedAt])
@@map("media_projects")
}
model MediaEditPlan {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
projectId String
project MediaProject @relation(fields: [projectId], references: [id], onDelete: Cascade)
profileId String
version Int
/// Edl, validated before insert.
edl Json
createdBy String
rationale String?
createdAt DateTime @default(now())
renders MediaRender[]
@@unique([projectId, profileId, version])
@@map("media_edit_plans")
}
model MediaRender {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
projectId String
project MediaProject @relation(fields: [projectId], references: [id], onDelete: Cascade)
planId String
plan MediaEditPlan @relation(fields: [planId], references: [id], onDelete: Cascade)
profileId String
quality String
status String @default("queued")
progress Float @default(0)
/// MediaRenderOutput[] plus internal relative paths.
outputs Json @default("[]")
gateReport Json?
ffprobe Json?
autoFixRound Int @default(0)
error String?
createdBy String
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
publications MediaPublication[]
@@index([projectId, createdAt])
@@index([spaceId, status])
@@map("media_renders")
}
model MediaPublication {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
renderId String
render MediaRender @relation(fields: [renderId], references: [id], onDelete: Cascade)
profileId String
destination String
status String @default("draft")
scheduledAt DateTime?
publishedAt DateTime?
externalUrl String?
targetUrl String?
trackingUrl String?
utm Json?
title String?
caption String?
tags String[] @default([])
thumbnailVariant String?
bundleArtifactId String?
createdBy String
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
metrics MediaMetric[]
@@index([spaceId, createdAt])
@@index([renderId])
@@map("media_publications")
}
model MediaMetric {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
publicationId String
publication MediaPublication @relation(fields: [publicationId], references: [id], onDelete: Cascade)
observedAt DateTime
views Int @default(0)
clicks Int @default(0)
conversions Int @default(0)
revenueCents Int @default(0)
currency String @default("EUR")
source String
createdAt DateTime @default(now())
@@unique([publicationId, observedAt, source])
@@index([spaceId, observedAt])
@@map("media_metrics")
}
model MediaMetricMapping {
id String @id @default(cuid())
spaceId String
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
name String
/// MetricCsvMapping without id
mapping Json
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@unique([spaceId, name])
@@map("media_metric_mappings")
}
model MediaTeaserModel {
spaceId String @id
space Space @relation(fields: [spaceId], references: [id], onDelete: Cascade)
/// TeaserWeights
weights Json
sampleCount Int
updatedAt DateTime @updatedAt
@@map("media_teaser_models")
}

View File

@ -7,6 +7,7 @@ export * from "./events.js";
export * from "./external-conversations.js";
export * from "./groups.js";
export * from "./main-agent.js";
export * from "./media.js";
export * from "./memory-config.js";
export * from "./messages.js";
export * from "./messaging.js";

View File

@ -0,0 +1,395 @@
import type { MediaCollection, MediaCollectionKind, MediaLabelSource } from "@rakazo/contracts";
import { roundCoordinate } from "@rakazo/media";
import type { MediaCollection as MediaCollectionRow, PrismaClient } from "./client.js";
import { Prisma } from "./client.js";
import type { CollectionStats, MediaDb, MediaPage, MediaPageInput } from "./media-shared.js";
import {
assertAssetsInSpace,
decodeKeysetCursor,
EMPTY_COLLECTION_STATS,
InvalidMediaInputError,
keysetAfterSql,
lockMediaKey,
mapMediaAsset,
mapMediaCollection,
pageLimit,
toPage,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
/** One session or theme cluster computed by the worker. */
export interface MediaClusterInput {
/** Stable key so a re-run updates the same collection. */
key: string;
title: string;
labelSource: Exclude<MediaLabelSource, "user">;
startAt: Date | null;
endAt: Date | null;
centroid: { lat: number; lon: number } | null;
coverAssetId: string | null;
/** In display order. */
assetIds: string[];
}
export interface ReplaceClustersResult {
created: number;
updated: number;
deleted: number;
}
async function collectionStats(db: MediaDb, ids: string[]): Promise<Map<string, CollectionStats>> {
if (ids.length === 0) return new Map();
const rows = await db.$queryRaw<
Array<{ id: string; assetCount: number; videoCount: number; totalDurationMs: bigint }>
>(Prisma.sql`
SELECT ci."collectionId" AS id,
count(*)::int AS "assetCount",
(count(*) FILTER (WHERE a.kind = 'video'))::int AS "videoCount",
coalesce(sum(a."durationMs"), 0)::bigint AS "totalDurationMs"
FROM media_collection_items ci
JOIN media_assets a ON a.id = ci."assetId"
WHERE ci."collectionId" IN (${Prisma.join(ids)})
GROUP BY ci."collectionId"
`);
return new Map(
rows.map((row) => [
row.id,
{
assetCount: row.assetCount,
videoCount: row.videoCount,
totalDurationMs: Number(row.totalDurationMs),
},
]),
);
}
async function collectionDto(db: MediaDb, row: MediaCollectionRow): Promise<MediaCollection> {
const stats = await collectionStats(db, [row.id]);
return mapMediaCollection(row, stats.get(row.id) ?? EMPTY_COLLECTION_STATS);
}
/** Assets ordered by capture time, undated last, with the span they cover. */
async function orderedAssets(db: MediaDb, spaceId: string, assetIds: string[]) {
const assets = await db.mediaAsset.findMany({
where: { spaceId, id: { in: [...new Set(assetIds)] } },
orderBy: [{ capturedAt: { sort: "asc", nulls: "last" } }, { id: "asc" }],
select: { id: true, capturedAt: true },
});
const dated = assets.flatMap((asset) => (asset.capturedAt ? [asset.capturedAt] : []));
return {
ids: assets.map((asset) => asset.id),
startAt: dated[0] ?? null,
endAt: dated.at(-1) ?? null,
};
}
async function writeItems(db: MediaDb, collectionId: string, assetIds: string[]): Promise<void> {
await db.mediaCollectionItem.deleteMany({ where: { collectionId } });
if (assetIds.length === 0) return;
await db.mediaCollectionItem.createMany({
data: [...new Set(assetIds)].map((assetId, position) => ({ collectionId, assetId, position })),
});
}
/** Sessions, themes and manual collections. */
export function createMediaCollectionRepos(prisma: PrismaClient) {
// -------------------------------------------------------------------------
// Collections
// -------------------------------------------------------------------------
async function requireCollection(db: MediaDb, spaceId: string, collectionId: string) {
const row = await db.mediaCollection.findFirst({ where: { id: collectionId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
/**
* Apply a full clustering run for one kind. Collections are matched by cluster key;
* a title the user set survives, stale rule/model clusters are removed, and manual
* collections are never touched.
*/
async function replaceClusters(
spaceId: string,
kind: Exclude<MediaCollectionKind, "manual">,
clusters: MediaClusterInput[],
): Promise<ReplaceClustersResult> {
const keys = new Set<string>();
for (const cluster of clusters) {
if (keys.has(cluster.key)) throw new InvalidMediaInputError("Duplicate cluster key");
keys.add(cluster.key);
if (cluster.coverAssetId && !cluster.assetIds.includes(cluster.coverAssetId)) {
throw new InvalidMediaInputError("Cover asset is not in its cluster");
}
}
return prisma.$transaction(
async (tx) => {
await lockMediaKey(tx, `clusters:${spaceId}:${kind}`);
await assertAssetsInSpace(
tx,
spaceId,
clusters.flatMap((cluster) => cluster.assetIds),
);
const existing = await tx.mediaCollection.findMany({
where: { spaceId, kind, clusterKey: { not: null } },
});
const byKey = new Map(existing.map((row) => [row.clusterKey as string, row]));
const result: ReplaceClustersResult = { created: 0, updated: 0, deleted: 0 };
for (const cluster of clusters) {
const fields = {
startAt: cluster.startAt,
endAt: cluster.endAt,
centroidLat: cluster.centroid ? roundCoordinate(cluster.centroid.lat) : null,
centroidLon: cluster.centroid ? roundCoordinate(cluster.centroid.lon) : null,
coverAssetId: cluster.coverAssetId,
};
const current = byKey.get(cluster.key);
let collectionId: string;
if (current) {
const keepTitle = current.labelSource === "user";
await tx.mediaCollection.update({
where: { id: current.id },
data: {
...fields,
...(keepTitle ? {} : { title: cluster.title, labelSource: cluster.labelSource }),
},
});
collectionId = current.id;
result.updated += 1;
} else {
const created = await tx.mediaCollection.create({
data: {
...fields,
spaceId,
kind,
clusterKey: cluster.key,
title: cluster.title,
labelSource: cluster.labelSource,
},
select: { id: true },
});
collectionId = created.id;
result.created += 1;
}
await writeItems(tx, collectionId, cluster.assetIds);
}
const stale = existing.filter(
(row) => !keys.has(row.clusterKey as string) && row.labelSource !== "user",
);
if (stale.length > 0) {
const deleted = await tx.mediaCollection.deleteMany({
where: { spaceId, id: { in: stale.map((row) => row.id) } },
});
result.deleted = deleted.count;
}
return result;
},
{ timeout: 120_000 },
);
}
async function listCollections(
spaceId: string,
input: MediaPageInput & { kind?: MediaCollectionKind },
): Promise<MediaPage<MediaCollection>> {
const limit = pageLimit(input.limit);
const cursor = input.cursor ? decodeKeysetCursor(input.cursor) : null;
const keys = await prisma.$queryRaw<Array<{ id: string }>>(Prisma.sql`
SELECT c.id FROM media_collections c
WHERE c."spaceId" = ${spaceId}
${input.kind ? Prisma.sql`AND c.kind = ${input.kind}` : Prisma.empty}
AND ${keysetAfterSql(cursor, Prisma.sql`c."startAt"`, Prisma.sql`c.id`)}
ORDER BY c."startAt" DESC NULLS LAST, c.id DESC
LIMIT ${limit + 1}
`);
const ids = keys.map((key) => key.id);
const [rows, stats] = await Promise.all([
prisma.mediaCollection.findMany({ where: { spaceId, id: { in: ids } } }),
collectionStats(prisma, ids),
]);
const byId = new Map(rows.map((row) => [row.id, row]));
const ordered = ids.flatMap((id) => {
const row = byId.get(id);
return row ? [row] : [];
});
return toPage(
ordered,
limit,
(row) => ({ at: row.startAt, id: row.id }),
(row) => mapMediaCollection(row, stats.get(row.id) ?? EMPTY_COLLECTION_STATS),
);
}
async function getCollection(spaceId: string, collectionId: string) {
const row = await requireCollection(prisma, spaceId, collectionId);
const items = await prisma.mediaCollectionItem.findMany({
where: { collectionId, asset: { spaceId } },
orderBy: [{ position: "asc" }, { assetId: "asc" }],
include: { asset: true },
});
return {
collection: await collectionDto(prisma, row),
assets: items.map((item) => mapMediaAsset(item.asset)),
};
}
/** Asset ids of a collection in display order. */
async function collectionAssetIds(spaceId: string, collectionIds: string[]): Promise<string[]> {
const ids = [...new Set(collectionIds)];
if (ids.length === 0) return [];
const owned = await prisma.mediaCollection.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
const items = await prisma.mediaCollectionItem.findMany({
where: { collectionId: { in: ids } },
orderBy: [{ position: "asc" }, { assetId: "asc" }],
select: { assetId: true, collectionId: true },
});
const order = new Map(ids.map((id, index) => [id, index]));
items.sort((a, b) => (order.get(a.collectionId) ?? 0) - (order.get(b.collectionId) ?? 0));
return [...new Set(items.map((item) => item.assetId))];
}
/** The user's title wins over every later clustering label. */
async function renameCollection(
spaceId: string,
collectionId: string,
title: string,
): Promise<MediaCollection> {
const result = await prisma.mediaCollection.updateMany({
where: { id: collectionId, spaceId },
data: { title, labelSource: "user" },
});
if (result.count !== 1) throw new IsolationError();
return collectionDto(prisma, await requireCollection(prisma, spaceId, collectionId));
}
/** Merge into the first collection; the others are deleted. */
async function mergeCollections(
spaceId: string,
collectionIds: string[],
): Promise<MediaCollection> {
const ids = [...new Set(collectionIds)];
if (ids.length < 2) throw new InvalidMediaInputError("Merge needs two collections");
return prisma.$transaction(async (tx) => {
const rows = await tx.mediaCollection.findMany({ where: { spaceId, id: { in: ids } } });
if (rows.length !== ids.length) throw new IsolationError();
const target = rows.find((row) => row.id === ids[0]) as MediaCollectionRow;
const items = await tx.mediaCollectionItem.findMany({
where: { collectionId: { in: ids } },
select: { assetId: true },
});
const assets = await orderedAssets(
tx,
spaceId,
items.map((item) => item.assetId),
);
await writeItems(tx, target.id, assets.ids);
await tx.mediaCollection.deleteMany({
where: { spaceId, id: { in: ids.filter((id) => id !== target.id) } },
});
const updated = await tx.mediaCollection.update({
where: { id: target.id },
data: {
startAt: assets.startAt,
endAt: assets.endAt,
labelSource: "user",
coverAssetId: target.coverAssetId ?? assets.ids[0] ?? null,
},
});
return collectionDto(tx, updated);
});
}
/** Split before `atAssetId`: it and every later asset move to a new collection. */
async function splitCollection(
spaceId: string,
collectionId: string,
atAssetId: string,
): Promise<MediaCollection[]> {
return prisma.$transaction(async (tx) => {
const row = await requireCollection(tx, spaceId, collectionId);
const items = await tx.mediaCollectionItem.findMany({
where: { collectionId },
orderBy: [{ position: "asc" }, { assetId: "asc" }],
select: { assetId: true },
});
const index = items.findIndex((item) => item.assetId === atAssetId);
if (index <= 0) throw new InvalidMediaInputError("Split asset must be inside the collection");
const first = await orderedAssets(
tx,
spaceId,
items.slice(0, index).map((item) => item.assetId),
);
const second = await orderedAssets(
tx,
spaceId,
items.slice(index).map((item) => item.assetId),
);
await writeItems(tx, collectionId, first.ids);
const kept = await tx.mediaCollection.update({
where: { id: collectionId },
data: {
startAt: first.startAt,
endAt: first.endAt,
labelSource: "user",
coverAssetId:
row.coverAssetId && first.ids.includes(row.coverAssetId)
? row.coverAssetId
: (first.ids[0] ?? null),
},
});
const created = await tx.mediaCollection.create({
data: {
spaceId,
kind: row.kind,
title: `${row.title} (2)`,
labelSource: "user",
startAt: second.startAt,
endAt: second.endAt,
centroidLat: row.centroidLat,
centroidLon: row.centroidLon,
coverAssetId:
row.coverAssetId && second.ids.includes(row.coverAssetId)
? row.coverAssetId
: (second.ids[0] ?? null),
},
});
await writeItems(tx, created.id, second.ids);
return [await collectionDto(tx, kept), await collectionDto(tx, created)];
});
}
async function createManualCollection(
spaceId: string,
title: string,
assetIds: string[],
): Promise<MediaCollection> {
return prisma.$transaction(async (tx) => {
await assertAssetsInSpace(tx, spaceId, assetIds);
const assets = await orderedAssets(tx, spaceId, assetIds);
const created = await tx.mediaCollection.create({
data: {
spaceId,
kind: "manual",
title,
labelSource: "user",
startAt: assets.startAt,
endAt: assets.endAt,
coverAssetId: assets.ids[0] ?? null,
},
});
await writeItems(tx, created.id, assets.ids);
return collectionDto(tx, created);
});
}
return {
replaceClusters,
listCollections,
getCollection,
collectionAssetIds,
renameCollection,
mergeCollections,
splitCollection,
createManualCollection,
};
}

View File

@ -0,0 +1,265 @@
import { randomUUID } from "node:crypto";
import type { MediaSourceItem } from "@rakazo/adapter-kit";
import type { BrandKit, TeaserWeights } from "@rakazo/contracts";
import { TEASER_FEATURES } from "@rakazo/contracts";
import { bootstrapUserSpace } from "./bootstrap-user.js";
import type { PrismaClient } from "./client.js";
import { createDb } from "./client.js";
import type { MediaRepos } from "./media.js";
/*
* Synthetic fixtures for the Media Studio Postgres tests. Nothing here describes a
* real person, place, or file.
*/
const signupEnv = { signupsEnabled: "true", signupAllowlist: "" };
export interface MediaTestDb {
prisma: PrismaClient;
newUser(): Promise<{ userId: string; spaceId: string }>;
cleanup(): Promise<void>;
}
export function openMediaTestDb(databaseUrl: string): MediaTestDb {
const db = createDb(databaseUrl, { poolMax: 8 });
const userIds: string[] = [];
return {
prisma: db.prisma,
async newUser() {
// bootstrapUserSpace derives the organization slug from the first 12 characters.
const id = `md${randomUUID().replaceAll("-", "")}`;
userIds.push(id);
await db.prisma.user.create({ data: { id, name: "Test", email: `${id}@rakazo.test` } });
const { spaceId } = await bootstrapUserSpace(db.prisma, { id }, signupEnv, {
claimDeploymentOwner: false,
});
return { userId: id, spaceId };
},
async cleanup() {
const memberships = await db.prisma.member.findMany({
where: { userId: { in: userIds } },
select: { organizationId: true },
});
await db.prisma.organization.deleteMany({
where: { id: { in: memberships.map((member) => member.organizationId) } },
});
await db.prisma.user.deleteMany({ where: { id: { in: userIds } } });
await db.prisma.$disconnect();
await db.pool.end();
},
};
}
export const BASE_TIME = Date.UTC(2026, 0, 1, 8, 0, 0);
export function sourceItem(
index: number,
overrides: Partial<MediaSourceItem> = {},
): MediaSourceItem {
return {
externalId: `clip-${String(index).padStart(5, "0")}.mp4`,
kind: index % 3 === 0 ? "photo" : "video",
capturedAt: new Date(BASE_TIME + index * 60_000),
durationMs: index % 3 === 0 ? null : 10_000,
width: 1920,
height: 1080,
fps: 30,
gpsLat: null,
gpsLon: null,
deviceModel: "Test Camera",
albumNames: [],
isFavorite: false,
fileName: `clip-${index}.mp4`,
sizeBytes: 1000 + index,
revision: "r1",
...overrides,
};
}
export function testEdl(assetId: string, profileId = "example-short-vertical") {
return {
edlVersion: 1,
output: {
profileId,
width: 1080,
height: 1920,
fps: 30,
durationTargetMs: 5000,
durationHardMinMs: 1000,
durationHardMaxMs: 60_000,
},
tracks: {
video: [{ id: "c1", assetId, inMs: 0, outMs: 5000, atMs: 0, reframe: "auto" }],
},
metadata: {},
};
}
export const testBrandKit: BrandKit = {
name: "Test Kit",
colors: {
primary: "#101010",
secondary: "#202020",
accent: "#303030",
text: "#f0f0f0",
background: "#000000",
},
fonts: [],
logoArtifactIds: [],
captionPresets: [],
musicBeds: [],
};
export const testTeaserWeights = Object.fromEntries(
TEASER_FEATURES.map((feature) => [feature, 1]),
) as TeaserWeights;
export function vectorBytes(values: number[]): Uint8Array {
return new Uint8Array(new Float32Array(values).buffer);
}
/** One of every media record in a space, linked the way the product links them. */
export async function seedMediaSpace(media: MediaRepos, spaceId: string, userId: string) {
const source = await media.createSource(spaceId, {
kind: "folder",
label: "Footage",
ownerUserId: userId,
config: { path: "/media/footage" },
});
const synced = await media.upsertSourceItems(spaceId, source.id, [
sourceItem(1, { albumNames: ["Holiday"] }),
sourceItem(2),
]);
const [assetId, secondAssetId] = synced.changedAssetIds as [string, string];
const analysis = await media.upsertAnalysis(spaceId, assetId, {
kind: "embeddings",
modelId: "test-embedder",
modelVersion: "1",
vectors: vectorBytes([1, 0, 0]),
});
await media.replaceSegments(spaceId, assetId, 1, [
{ kind: "shot", startMs: 0, endMs: 4000, labels: ["beach"], transcriptExcerpt: "hello" },
]);
const segments = await media.listSegments(spaceId, [assetId]);
const tracks = await media.replaceFaceTracks(spaceId, assetId, "faces-1", [
{
startMs: 0,
endMs: 3000,
confidenceMin: 0.9,
keyframesPath: `faces/${assetId}/t1.json`,
embedding: vectorBytes([0.1, 0.2, 0.3]),
},
]);
const [person] = await media.createPersonSuggestions(spaceId, [
{ centroid: vectorBytes([0.1, 0.2, 0.3]), trackIds: tracks.map((track) => track.id) },
]);
const mark = await media.createMark(spaceId, {
kind: "tattoo",
label: "arm",
personId: person?.id ?? null,
assetId,
prompt: { tMs: 1000, point: { x: 0.5, y: 0.5 } },
treatment: "blur",
});
await media.replaceClusters(spaceId, "session", [
{
key: "s1",
title: "Morning",
labelSource: "rule",
startAt: new Date(BASE_TIME),
endAt: new Date(BASE_TIME + 120_000),
centroid: null,
coverAssetId: assetId,
assetIds: [assetId, secondAssetId],
},
]);
const collection = (await media.listCollections(spaceId, {})).items[0];
const brandKit = await media.upsertBrandKit(spaceId, { kit: testBrandKit });
const project = await media.createProject(spaceId, userId, {
title: "Teaser",
goal: "teaser",
profileIds: ["example-short-vertical"],
collectionIds: collection ? [collection.id] : [],
brandKitId: brandKit.id,
});
const plan = await media.createPlanVersion(spaceId, {
projectId: project.id,
profileId: "example-short-vertical",
edl: testEdl(assetId),
createdBy: "agent",
rationale: null,
});
const render = await media.createRender(spaceId, {
planId: plan.id,
quality: "draft",
createdBy: userId,
});
await media.claimRender(spaceId, render.id);
await media.updateRender(spaceId, render.id, { status: "done", progress: 1 });
const queuedRender = await media.createRender(spaceId, {
planId: plan.id,
quality: "final",
createdBy: userId,
});
const publication = await media.createPublication(spaceId, userId, {
renderId: render.id,
destination: "bundle",
targetUrl: "https://example.com/offer",
trackingUrl: "https://example.com/offer?utm_content=a",
utm: { source: "shorts", medium: "video", campaign: "launch", content: "a" },
title: "Title",
caption: "Caption",
tags: ["test"],
thumbnailVariant: "t1",
scheduledAt: null,
});
const metric = await media.recordMetric(spaceId, {
publicationId: publication.id,
observedAt: new Date(BASE_TIME).toISOString(),
views: 100,
clicks: 10,
conversions: 1,
revenueCents: 999,
currency: "EUR",
source: "manual",
});
const mapping = await media.saveMetricMapping(spaceId, {
name: "Default",
delimiter: ",",
columns: { publication: "id", observedAt: "date", views: "views" },
publicationKey: "id",
revenueUnit: "major",
defaultCurrency: "EUR",
});
await media.putTeaserModel(spaceId, testTeaserWeights, 3);
const upload = await media.beginUpload(spaceId, {
assetId: secondAssetId,
userId,
sizeBytes: 10,
sha256: "a".repeat(64),
fileName: "clip.mp4",
chunkBytes: 4,
});
return {
sourceId: source.id,
assetId,
secondAssetId,
analysisId: analysis.id,
segmentId: segments[0]?.id as string,
trackId: tracks[0]?.id as string,
personId: person?.id as string,
markId: mark.id,
collectionId: collection?.id as string,
brandKitId: brandKit.id,
projectId: project.id,
planId: plan.id,
renderId: render.id,
queuedRenderId: queuedRender.id,
publicationId: publication.id,
metricId: metric.id,
mappingId: mapping.id as string,
uploadId: upload.id,
};
}
export type SeededMediaSpace = Awaited<ReturnType<typeof seedMediaSpace>>;

View File

@ -0,0 +1,428 @@
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import type { MediaRepos } from "./media.js";
import { createMediaRepos } from "./media.js";
import type { MediaTestDb, SeededMediaSpace } from "./media-fixtures.js";
import {
openMediaTestDb,
seedMediaSpace,
sourceItem,
testBrandKit,
testEdl,
vectorBytes,
} from "./media-fixtures.js";
import { IsolationError } from "./scope.js";
const databaseUrl = process.env.DATABASE_URL;
const describePostgres =
process.env.VERIFY_DATABASE && databaseUrl ? describe.sequential : describe.skip;
describePostgres("media repositories: space isolation (PostgreSQL)", () => {
let db: MediaTestDb;
let media: MediaRepos;
let a: SeededMediaSpace;
let owner: { userId: string; spaceId: string };
let intruder: { userId: string; spaceId: string };
beforeAll(async () => {
db = openMediaTestDb(databaseUrl!);
media = createMediaRepos(db.prisma);
owner = await db.newUser();
intruder = await db.newUser();
a = await seedMediaSpace(media, owner.spaceId, owner.userId);
});
afterAll(async () => {
await db?.cleanup();
});
async function snapshot() {
const where = { spaceId: owner.spaceId };
const p = db.prisma;
return {
sources: await p.mediaSource.findMany({ where, orderBy: { id: "asc" } }),
assets: await p.mediaAsset.findMany({ where, orderBy: { id: "asc" } }),
segments: await p.mediaSegment.findMany({ where, orderBy: { id: "asc" } }),
analyses: await p.mediaAnalysis.findMany({ where, orderBy: { id: "asc" } }),
collections: await p.mediaCollection.findMany({ where, orderBy: { id: "asc" } }),
items: await p.mediaCollectionItem.findMany({
where: { collection: where },
orderBy: [{ collectionId: "asc" }, { assetId: "asc" }],
}),
people: await p.mediaPerson.findMany({ where, orderBy: { id: "asc" } }),
tracks: await p.mediaFaceTrack.findMany({ where, orderBy: { id: "asc" } }),
marks: await p.mediaMark.findMany({ where, orderBy: { id: "asc" } }),
kits: await p.mediaBrandKit.findMany({ where, orderBy: { id: "asc" } }),
projects: await p.mediaProject.findMany({ where, orderBy: { id: "asc" } }),
plans: await p.mediaEditPlan.findMany({ where, orderBy: { id: "asc" } }),
renders: await p.mediaRender.findMany({ where, orderBy: { id: "asc" } }),
publications: await p.mediaPublication.findMany({ where, orderBy: { id: "asc" } }),
metrics: await p.mediaMetric.findMany({ where, orderBy: { id: "asc" } }),
mappings: await p.mediaMetricMapping.findMany({ where, orderBy: { id: "asc" } }),
teaser: await p.mediaTeaserModel.findMany({ where }),
uploads: await p.mediaUpload.findMany({ where, orderBy: { id: "asc" } }),
};
}
it("rejects every read and write of another space's ids and leaves its rows untouched", async () => {
const before = await snapshot();
const b = intruder.spaceId;
const denied: Array<[string, () => Promise<unknown>]> = [
["getSource", () => media.getSource(b, a.sourceId)],
["updateSourceSync", () => media.updateSourceSync(b, a.sourceId, { status: "error" })],
["removeSource", () => media.removeSource(b, a.sourceId)],
["upsertSourceItems", () => media.upsertSourceItems(b, a.sourceId, [sourceItem(99)])],
["getAsset", () => media.getAsset(b, a.assetId)],
["getAssetRecord", () => media.getAssetRecord(b, a.assetId)],
[
"transitionIngestState",
() => media.transitionIngestState(b, a.assetId, { from: ["indexed"], to: "proxying" }),
],
[
"recordProxy",
() => media.recordProxy(b, a.assetId, { contentHash: "x", proxyPath: "p", thumbPath: "t" }),
],
["recordAnalysisVersion", () => media.recordAnalysisVersion(b, a.assetId, 9, 1)],
["requestUpload", () => media.requestUpload(b, [a.assetId])],
[
"upsertAnalysis",
() =>
media.upsertAnalysis(b, a.assetId, {
kind: "embeddings",
modelId: "m",
modelVersion: "1",
vectors: vectorBytes([9]),
}),
],
["latestAnalysis", () => media.latestAnalysis(b, a.assetId, "embeddings")],
[
"replaceSegments",
() => media.replaceSegments(b, a.assetId, 2, [{ kind: "shot", startMs: 0, endMs: 1 }]),
],
["setSegmentsPaidOnly", () => media.setSegmentsPaidOnly(b, [a.segmentId], true)],
[
"beginUpload",
() =>
media.beginUpload(b, {
assetId: a.assetId,
userId: intruder.userId,
sizeBytes: 1,
sha256: "b".repeat(64),
fileName: "x",
}),
],
["uploadStatus", () => media.uploadStatus(b, a.uploadId)],
["getUploadRecord", () => media.getUploadRecord(b, a.uploadId)],
["markChunkReceived", () => media.markChunkReceived(b, a.uploadId, 0)],
["completeUpload", () => media.completeUpload(b, a.uploadId, { storagePath: "x" })],
["failUpload", () => media.failUpload(b, a.uploadId)],
[
"replaceClusters",
() =>
media.replaceClusters(b, "session", [
{
key: "s1",
title: "Stolen",
labelSource: "rule",
startAt: null,
endAt: null,
centroid: null,
coverAssetId: null,
assetIds: [a.assetId],
},
]),
],
["getCollection", () => media.getCollection(b, a.collectionId)],
["collectionAssetIds", () => media.collectionAssetIds(b, [a.collectionId])],
["renameCollection", () => media.renameCollection(b, a.collectionId, "Mine")],
["mergeCollections", () => media.mergeCollections(b, [a.collectionId, `${a.collectionId}x`])],
["splitCollection", () => media.splitCollection(b, a.collectionId, a.secondAssetId)],
["createManualCollection", () => media.createManualCollection(b, "Mine", [a.assetId])],
["getPerson", () => media.getPerson(b, a.personId)],
[
"createPersonSuggestions",
() =>
media.createPersonSuggestions(b, [{ centroid: vectorBytes([1]), trackIds: [a.trackId] }]),
],
["updatePerson", () => media.updatePerson(b, a.personId, { displayName: "Mine" })],
["mergePeople", () => media.mergePeople(b, [a.personId, `${a.personId}x`])],
["assignTrack", () => media.assignTrack(b, a.trackId, null)],
[
"recordConsent (foreign person)",
() =>
media.recordConsent(
{ spaceId: b, recordedByUserId: intruder.userId },
{
personId: a.personId,
consentArtifactId: null,
consentRecordedAt: new Date().toISOString(),
ageVerificationMethod: "none",
ageVerifiedAt: null,
},
),
],
[
"recordConsent (foreign recorder)",
() =>
media.recordConsent(
{ spaceId: owner.spaceId, recordedByUserId: intruder.userId },
{
personId: a.personId,
consentArtifactId: null,
consentRecordedAt: new Date().toISOString(),
ageVerificationMethod: "none",
ageVerifiedAt: null,
},
),
],
["removePersonCompletely", () => media.removePersonCompletely(b, a.personId)],
["replaceFaceTracks", () => media.replaceFaceTracks(b, a.assetId, "faces-2", [])],
[
"createMark",
() =>
media.createMark(b, {
kind: "scar",
label: null,
personId: null,
assetId: a.assetId,
prompt: { tMs: 0 },
treatment: "blur",
}),
],
["getMark", () => media.getMark(b, a.markId)],
["updateMark", () => media.updateMark(b, a.markId, { treatment: "cover" })],
[
"recordMarkResult",
() =>
media.recordMarkResult(b, a.markId, {
qualityScore: 0,
suggestedFallback: "cover",
maskDir: "x",
}),
],
["removeMark", () => media.removeMark(b, a.markId)],
["upsertBrandKit", () => media.upsertBrandKit(b, { id: a.brandKitId, kit: testBrandKit })],
["getBrandKit", () => media.getBrandKit(b, a.brandKitId)],
["removeBrandKit", () => media.removeBrandKit(b, a.brandKitId)],
[
"createProject (foreign brand kit)",
() =>
media.createProject(b, intruder.userId, {
title: "x",
goal: "teaser",
profileIds: ["example-short-vertical"],
brandKitId: a.brandKitId,
}),
],
[
"createProject (foreign collection)",
() =>
media.createProject(b, intruder.userId, {
title: "x",
goal: "teaser",
profileIds: ["example-short-vertical"],
collectionIds: [a.collectionId],
}),
],
["updateProject", () => media.updateProject(b, a.projectId, { title: "Mine" })],
["getProject", () => media.getProject(b, a.projectId)],
["getProjectRecord", () => media.getProjectRecord(b, a.projectId)],
["removeProject", () => media.removeProject(b, a.projectId)],
[
"createPlanVersion",
() =>
media.createPlanVersion(b, {
projectId: a.projectId,
profileId: "example-short-vertical",
edl: testEdl(a.assetId),
createdBy: "agent",
rationale: null,
}),
],
["getPlan", () => media.getPlan(b, a.planId)],
["latestPlan", () => media.latestPlan(b, a.projectId, "example-short-vertical")],
["listPlanVersions", () => media.listPlanVersions(b, a.projectId, "example-short-vertical")],
[
"createRender",
() => media.createRender(b, { planId: a.planId, quality: "draft", createdBy: "x" }),
],
["getRender", () => media.getRender(b, a.renderId)],
["getRenderRecord", () => media.getRenderRecord(b, a.renderId)],
["claimRender", () => media.claimRender(b, a.queuedRenderId)],
["updateRender", () => media.updateRender(b, a.queuedRenderId, { progress: 0.5 })],
["cancelRender", () => media.cancelRender(b, a.queuedRenderId)],
[
"createPublication",
() =>
media.createPublication(b, intruder.userId, {
renderId: a.renderId,
destination: "bundle",
targetUrl: null,
trackingUrl: null,
utm: null,
title: null,
caption: null,
tags: [],
thumbnailVariant: null,
scheduledAt: null,
}),
],
["getPublication", () => media.getPublication(b, a.publicationId)],
["getPublicationRecord", () => media.getPublicationRecord(b, a.publicationId)],
["schedulePublication", () => media.schedulePublication(b, a.publicationId, new Date())],
["cancelPublication", () => media.cancelPublication(b, a.publicationId)],
["markPublicationBundled", () => media.markPublicationBundled(b, a.publicationId, null)],
["markPublicationPublished", () => media.markPublicationPublished(b, a.publicationId, null)],
["markPublicationFailed", () => media.markPublicationFailed(b, a.publicationId)],
[
"recordMetric",
() =>
media.recordMetric(b, {
publicationId: a.publicationId,
observedAt: new Date().toISOString(),
views: 1,
clicks: 0,
conversions: 0,
revenueCents: 0,
currency: "EUR",
source: "manual",
}),
],
[
"recordMetrics",
() =>
media.recordMetrics(b, [
{
publicationId: a.publicationId,
observedAt: new Date().toISOString(),
views: 1,
clicks: 0,
conversions: 0,
revenueCents: 0,
currency: "EUR",
source: "csv_import",
},
]),
],
["listMetrics", () => media.listMetrics(b, a.publicationId)],
[
"saveMetricMapping (foreign id)",
() =>
media.saveMetricMapping(b, {
id: a.mappingId,
name: "Stolen",
delimiter: ",",
columns: { publication: "id", observedAt: "date" },
publicationKey: "id",
revenueUnit: "major",
defaultCurrency: "EUR",
}),
],
["getMetricMapping", () => media.getMetricMapping(b, a.mappingId)],
["funnel (foreign project)", () => media.funnel(b, a.projectId)],
];
for (const [name, call] of denied) {
await expect(call(), name).rejects.toBeInstanceOf(IsolationError);
}
// Reads without ids return nothing of the other space.
expect(await media.listSources(b)).toEqual([]);
expect((await media.searchAssets(b, {})).items).toEqual([]);
expect((await media.searchAssets(b, { text: "clip" })).items).toEqual([]);
expect((await media.searchAssets(b, { personId: a.personId })).items).toEqual([]);
expect((await media.searchAssets(b, { collectionId: a.collectionId })).items).toEqual([]);
expect((await media.searchAssets(b, { sourceId: a.sourceId })).items).toEqual([]);
expect(await media.listAssetRecords(b, [a.assetId])).toEqual([]);
expect(await media.listAssetsForClustering(b)).toEqual([]);
expect(await media.assetsNeedingWork(b, "indexed", 10)).toEqual([]);
expect((await media.latestAnalysesByKind(b, "embeddings")).size).toBe(0);
expect(await media.listSegments(b, [a.assetId])).toEqual([]);
expect((await media.listCollections(b, {})).items).toEqual([]);
expect(await media.listPeople(b)).toEqual([]);
expect(await media.listPersonRecords(b, [a.personId])).toEqual([]);
expect(await media.personCentroids(b)).toEqual([]);
expect(await media.listFaceTracks(b, { assetId: a.assetId })).toEqual([]);
expect(await media.faceTrackRecords(b, [a.assetId])).toEqual([]);
expect(await media.faceTrackEmbeddings(b)).toEqual([]);
expect(await media.listMarks(b, { assetId: a.assetId })).toEqual([]);
expect(await media.marksForAsset(b, a.assetId)).toEqual([]);
expect(await media.listBrandKits(b)).toEqual([]);
expect((await media.listProjects(b)).items).toEqual([]);
expect(await media.latestPlans(b, a.projectId)).toEqual([]);
expect(await media.listRenders(b, a.projectId)).toEqual([]);
expect((await media.listPublications(b, { projectId: a.projectId })).items).toEqual([]);
expect((await media.resolvePublicationKeys(b, "id", [a.publicationId])).size).toBe(0);
expect(await media.listMetricMappings(b)).toEqual([]);
expect(await media.funnel(b)).toEqual([]);
expect(await media.publicationsWithMetricsCount(b)).toBe(0);
expect(await media.getTeaserModel(b)).toBeNull();
await media.resetTeaserModel(b);
expect(await snapshot()).toEqual(before);
// The owner still sees everything.
expect((await media.searchAssets(owner.spaceId, {})).items).toHaveLength(2);
expect(await media.getTeaserModel(owner.spaceId)).not.toBeNull();
});
it("resolves the owning space for worker jobs", async () => {
expect(await media.mediaSpaceOf("source", a.sourceId)).toBe(owner.spaceId);
expect(await media.mediaSpaceOf("asset", a.assetId)).toBe(owner.spaceId);
expect(await media.mediaSpaceOf("render", a.renderId)).toBe(owner.spaceId);
expect(await media.mediaSpaceOf("publication", a.publicationId)).toBe(owner.spaceId);
expect(await media.mediaSpaceOf("asset", "missing")).toBeNull();
});
it("deleting a space removes every media row of it and nothing else", async () => {
const other = await db.newUser();
const seeded = await seedMediaSpace(media, other.spaceId, other.userId);
expect(seeded.assetId).toBeTruthy();
const tables = [
"media_sources",
"media_assets",
"media_uploads",
"media_segments",
"media_analyses",
"media_collections",
"media_people",
"media_face_tracks",
"media_marks",
"media_brand_kits",
"media_projects",
"media_edit_plans",
"media_renders",
"media_publications",
"media_metrics",
"media_metric_mappings",
"media_teaser_models",
];
async function counts(spaceId: string) {
const out: Record<string, number> = {};
for (const table of tables) {
const rows = await db.prisma.$queryRawUnsafe<Array<{ count: number }>>(
`SELECT count(*)::int AS count FROM ${table} WHERE "spaceId" = $1`,
spaceId,
);
out[table] = rows[0]?.count ?? -1;
}
const items = await db.prisma.$queryRawUnsafe<Array<{ count: number }>>(
`SELECT count(*)::int AS count FROM media_collection_items ci
JOIN media_assets a ON a.id = ci."assetId" WHERE a."spaceId" = $1`,
spaceId,
);
out.media_collection_items = items[0]?.count ?? -1;
return out;
}
const populated = await counts(other.spaceId);
for (const table of tables) expect(populated[table], table).toBeGreaterThan(0);
const ownerBefore = await counts(owner.spaceId);
const itemCountBefore = await db.prisma.mediaCollectionItem.count();
await db.prisma.space.delete({ where: { id: other.spaceId } });
const after = await counts(other.spaceId);
for (const [table, count] of Object.entries(after)) expect(count, table).toBe(0);
expect(await db.prisma.mediaCollectionItem.count()).toBe(
itemCountBefore - (populated.media_collection_items ?? 0),
);
expect(await counts(owner.spaceId)).toEqual(ownerBefore);
});
});

View File

@ -0,0 +1,800 @@
import type { MediaSourceItem } from "@rakazo/adapter-kit";
import type {
MediaAnalysisKind,
MediaAsset,
MediaAssetQuery,
MediaFaceTrack,
MediaIngestState,
MediaSegment,
MediaSegmentKind,
MediaSource,
MediaSourceKind,
MediaSourceStatus,
TeaserFeature,
} from "@rakazo/contracts";
import { CLIENT_PUSHED_MEDIA_SOURCES } from "@rakazo/contracts";
import { roundCoordinate } from "@rakazo/media";
import type {
MediaAnalysis as MediaAnalysisRow,
MediaAsset as MediaAssetRow,
MediaSource as MediaSourceRow,
PrismaClient,
} from "./client.js";
import { Prisma } from "./client.js";
import type { MediaPage, MediaPageInput } from "./media-shared.js";
import {
decodeKeysetCursor,
InvalidMediaInputError,
keysetAfterSql,
likeContains,
lockMediaKey,
mapMediaAsset,
mapMediaFaceTrack,
mapMediaSegment,
mapMediaSource,
pageLimit,
toBytes,
toPage,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
export interface NewMediaSource {
kind: MediaSourceKind;
label: string;
ownerUserId: string;
/** Folder: { path }. Client-pushed: { deviceId }. Never secrets. */
config: Record<string, unknown>;
}
/** Source row for the sync worker, including its config and cursor. */
export interface MediaSourceRecord extends MediaSource {
spaceId: string;
ownerUserId: string;
config: Record<string, unknown>;
lastSyncCursor: string | null;
}
export interface MediaSourceSyncPatch {
status?: MediaSourceStatus;
lastSyncCursor?: string | null;
lastSyncedAt?: Date | null;
error?: string | null;
}
export interface UpsertSourceItemsResult {
created: number;
updated: number;
unchanged: number;
/** New assets and assets whose revision changed, in input order. */
changedAssetIds: string[];
}
export type MediaAssetSearch = Omit<MediaAssetQuery, "cursor" | "limit"> & MediaPageInput;
export interface MediaAssetDetail {
asset: MediaAsset;
segments: MediaSegment[];
faceTracks: MediaFaceTrack[];
transcript: string | null;
}
export interface MediaProxyResult {
contentHash: string;
proxyPath: string | null;
thumbPath: string | null;
spritePath?: string | null;
durationMs?: number | null;
width?: number | null;
height?: number | null;
fps?: number | null;
codec?: string | null;
orientation?: number | null;
}
export interface NewMediaAnalysis {
kind: MediaAnalysisKind;
modelId: string;
modelVersion: string;
data?: Prisma.InputJsonValue | null;
dataPath?: string | null;
/** Float32 little-endian vectors (see encodeVectors in @rakazo/media). */
vectors?: Uint8Array | null;
}
export interface NewMediaSegment {
kind: MediaSegmentKind;
startMs: number;
endMs: number;
scores?: Partial<Record<TeaserFeature, number>>;
teaserScore?: number | null;
transcriptExcerpt?: string | null;
labels?: string[];
paidOnly?: boolean;
}
export type MediaSpaceLookup = "source" | "asset" | "render" | "publication";
const EARTH_RADIUS_KM = 6371.0088;
const KM_PER_DEGREE = 111.32;
function sourceRecord(row: MediaSourceRow, assetCount: number): MediaSourceRecord {
const config =
row.config && typeof row.config === "object" && !Array.isArray(row.config)
? (row.config as Record<string, unknown>)
: {};
return {
...mapMediaSource(row, assetCount),
spaceId: row.spaceId,
ownerUserId: row.ownerUserId,
config,
lastSyncCursor: row.lastSyncCursor,
};
}
function rounded(value: number | null): number | null {
return value === null ? null : roundCoordinate(value);
}
function itemData(item: MediaSourceItem) {
return {
kind: item.kind,
revision: item.revision,
capturedAt: item.capturedAt,
durationMs: item.durationMs,
width: item.width,
height: item.height,
fps: item.fps,
gpsLat: rounded(item.gpsLat),
gpsLon: rounded(item.gpsLon),
deviceModel: item.deviceModel,
albumNames: item.albumNames,
isFavorite: item.isFavorite,
fileName: item.fileName,
sizeBytes: item.sizeBytes === null ? null : BigInt(item.sizeBytes),
};
}
/** Plain text of a stored transcript analysis (TranscribeResult words or a text field). */
function transcriptText(data: Prisma.JsonValue | null): string | null {
if (!data || typeof data !== "object" || Array.isArray(data)) return null;
if (typeof data.text === "string") return data.text;
if (!Array.isArray(data.words)) return null;
const words = data.words
.map((word) =>
word && typeof word === "object" && !Array.isArray(word) && typeof word.text === "string"
? word.text.trim()
: "",
)
.filter(Boolean);
return words.length > 0 ? words.join(" ") : null;
}
function nearSql(near: NonNullable<MediaAssetQuery["near"]>): Prisma.Sql {
const { lat, lon, radiusKm } = near;
const dLat = radiusKm / KM_PER_DEGREE;
const conditions: Prisma.Sql[] = [
Prisma.sql`a."gpsLat" IS NOT NULL AND a."gpsLon" IS NOT NULL`,
Prisma.sql`a."gpsLat" BETWEEN ${lat - dLat}::float8 AND ${lat + dLat}::float8`,
];
// Longitude prefilter only where it is a simple band; near the poles the haversine alone decides.
const maxAbsLat = Math.min(90, Math.abs(lat) + dLat);
const cos = Math.cos((maxAbsLat * Math.PI) / 180);
if (cos > 0.01) {
const dLon = radiusKm / (KM_PER_DEGREE * cos);
if (dLon < 180) {
const west = lon - dLon;
const east = lon + dLon;
if (west < -180) {
conditions.push(
Prisma.sql`(a."gpsLon" >= ${west + 360}::float8 OR a."gpsLon" <= ${east}::float8)`,
);
} else if (east > 180) {
conditions.push(
Prisma.sql`(a."gpsLon" >= ${west}::float8 OR a."gpsLon" <= ${east - 360}::float8)`,
);
} else {
conditions.push(Prisma.sql`a."gpsLon" BETWEEN ${west}::float8 AND ${east}::float8`);
}
}
}
conditions.push(Prisma.sql`
2 * ${EARTH_RADIUS_KM}::float8 * asin(least(1, sqrt(
power(sin(radians(a."gpsLat" - ${lat}::float8) / 2), 2)
+ cos(radians(${lat}::float8)) * cos(radians(a."gpsLat"))
* power(sin(radians(a."gpsLon" - ${lon}::float8) / 2), 2)
))) <= ${radiusKm}::float8`);
return Prisma.join(conditions, " AND ");
}
function textSql(spaceId: string, text: string): Prisma.Sql {
const pattern = likeContains(text);
return Prisma.sql`(
a."fileName" ILIKE ${pattern}
OR EXISTS (SELECT 1 FROM unnest(a."albumNames") AS album(name) WHERE album.name ILIKE ${pattern})
OR EXISTS (
SELECT 1 FROM media_collection_items ci
JOIN media_collections c ON c.id = ci."collectionId"
WHERE ci."assetId" = a.id AND c."spaceId" = ${spaceId} AND c.title ILIKE ${pattern}
)
OR EXISTS (
SELECT 1 FROM media_segments s
WHERE s."assetId" = a.id AND s."spaceId" = ${spaceId}
AND (
s."transcriptExcerpt" ILIKE ${pattern}
OR EXISTS (SELECT 1 FROM unnest(s.labels) AS label(name) WHERE label.name ILIKE ${pattern})
)
)
)`;
}
export function createMediaLibraryRepos(prisma: PrismaClient) {
async function requireAsset(
db: Prisma.TransactionClient | PrismaClient,
spaceId: string,
assetId: string,
): Promise<MediaAssetRow> {
const asset = await db.mediaAsset.findFirst({ where: { id: assetId, spaceId } });
if (!asset) throw new IsolationError();
return asset;
}
async function updateAsset(
spaceId: string,
assetId: string,
data: Prisma.MediaAssetUpdateManyMutationInput,
): Promise<MediaAsset> {
const result = await prisma.mediaAsset.updateMany({ where: { id: assetId, spaceId }, data });
if (result.count !== 1) throw new IsolationError();
return mapMediaAsset(await requireAsset(prisma, spaceId, assetId));
}
// -------------------------------------------------------------------------
// Sources
// -------------------------------------------------------------------------
async function createSource(spaceId: string, input: NewMediaSource): Promise<MediaSource> {
const row = await prisma.mediaSource.create({
data: {
spaceId,
kind: input.kind,
label: input.label,
ownerUserId: input.ownerUserId,
config: input.config as Prisma.InputJsonObject,
},
});
return mapMediaSource(row, 0);
}
async function listSources(spaceId: string): Promise<MediaSource[]> {
const rows = await prisma.mediaSource.findMany({
where: { spaceId },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
include: { _count: { select: { assets: true } } },
});
return rows.map((row) => mapMediaSource(row, row._count.assets));
}
async function getSource(spaceId: string, sourceId: string): Promise<MediaSourceRecord> {
const row = await prisma.mediaSource.findFirst({
where: { id: sourceId, spaceId },
include: { _count: { select: { assets: true } } },
});
if (!row) throw new IsolationError();
return sourceRecord(row, row._count.assets);
}
async function updateSourceSync(
spaceId: string,
sourceId: string,
patch: MediaSourceSyncPatch,
): Promise<MediaSourceRecord> {
const result = await prisma.mediaSource.updateMany({
where: { id: sourceId, spaceId },
data: {
...(patch.status !== undefined ? { status: patch.status } : {}),
...(patch.lastSyncCursor !== undefined ? { lastSyncCursor: patch.lastSyncCursor } : {}),
...(patch.lastSyncedAt !== undefined ? { lastSyncedAt: patch.lastSyncedAt } : {}),
...(patch.error !== undefined ? { error: patch.error } : {}),
},
});
if (result.count !== 1) throw new IsolationError();
return getSource(spaceId, sourceId);
}
/** Deletes the source and, by cascade, its assets and everything derived from them. */
async function removeSource(spaceId: string, sourceId: string): Promise<void> {
const result = await prisma.mediaSource.deleteMany({ where: { id: sourceId, spaceId } });
if (result.count !== 1) throw new IsolationError();
}
/**
* Delete assets a device reported as removed. Returns the media-root-relative files
* the caller must delete (originals, proxies, thumbnails, face keyframes, masks).
*/
async function removeSourceItems(
spaceId: string,
sourceId: string,
externalIds: string[],
): Promise<{ deleted: number; paths: string[] }> {
if (externalIds.length === 0) return { deleted: 0, paths: [] };
return prisma.$transaction(async (tx) => {
const source = await tx.mediaSource.findFirst({ where: { id: sourceId, spaceId } });
if (!source) throw new IsolationError();
const assets = await tx.mediaAsset.findMany({
where: { spaceId, sourceId, externalId: { in: externalIds } },
select: {
id: true,
storagePath: true,
proxyPath: true,
thumbPath: true,
spritePath: true,
faceTracks: { select: { keyframesPath: true } },
marks: { select: { maskDir: true } },
},
});
const paths = assets.flatMap((asset) => [
...[asset.storagePath, asset.proxyPath, asset.thumbPath, asset.spritePath],
...asset.faceTracks.map((track) => track.keyframesPath),
...asset.marks.map((mark) => mark.maskDir),
]);
const result = await tx.mediaAsset.deleteMany({
where: { spaceId, id: { in: assets.map((asset) => asset.id) } },
});
return {
deleted: result.count,
paths: paths.filter((path): path is string => typeof path === "string" && path !== ""),
};
});
}
/**
* Worker entry point: jobs carry only a record id. Returns the owning space so every
* further call stays scoped, or null when the record no longer exists.
*/
async function mediaSpaceOf(kind: MediaSpaceLookup, id: string): Promise<string | null> {
const select = { spaceId: true } as const;
const row =
kind === "source"
? await prisma.mediaSource.findUnique({ where: { id }, select })
: kind === "asset"
? await prisma.mediaAsset.findUnique({ where: { id }, select })
: kind === "render"
? await prisma.mediaRender.findUnique({ where: { id }, select })
: await prisma.mediaPublication.findUnique({ where: { id }, select });
return row?.spaceId ?? null;
}
// -------------------------------------------------------------------------
// Assets
// -------------------------------------------------------------------------
/**
* Index one page of source items keyed by (sourceId, externalId). An item whose
* revision matches the stored one is unchanged; syncs of the same source are serialized.
*/
async function upsertSourceItems(
spaceId: string,
sourceId: string,
items: MediaSourceItem[],
): Promise<UpsertSourceItemsResult> {
const unique = new Map<string, MediaSourceItem>();
for (const item of items) unique.set(item.externalId, item);
return prisma.$transaction(
async (tx) => {
const source = await tx.mediaSource.findFirst({
where: { id: sourceId, spaceId },
select: { kind: true },
});
if (!source) throw new IsolationError();
await lockMediaKey(tx, `source:${sourceId}`);
const storage = (CLIENT_PUSHED_MEDIA_SOURCES as readonly string[]).includes(source.kind)
? "remote"
: "referenced";
const existing = await tx.mediaAsset.findMany({
where: { sourceId, externalId: { in: [...unique.keys()] } },
select: { id: true, externalId: true, revision: true },
});
const byExternalId = new Map(existing.map((row) => [row.externalId, row]));
const toCreate: MediaSourceItem[] = [];
const changed = new Map<string, string>();
let unchanged = 0;
for (const item of unique.values()) {
const current = byExternalId.get(item.externalId);
if (!current) {
toCreate.push(item);
} else if (current.revision === item.revision) {
unchanged += 1;
} else {
await tx.mediaAsset.update({ where: { id: current.id }, data: itemData(item) });
changed.set(item.externalId, current.id);
}
}
const updated = changed.size;
if (toCreate.length > 0) {
const created = await tx.mediaAsset.createManyAndReturn({
data: toCreate.map((item) => ({
...itemData(item),
spaceId,
sourceId,
externalId: item.externalId,
storage,
ingestState: storage === "remote" ? "awaiting_upload" : "indexed",
})),
select: { id: true, externalId: true },
});
for (const row of created) changed.set(row.externalId, row.id);
}
return {
created: toCreate.length,
updated,
unchanged,
changedAssetIds: [...unique.keys()].flatMap((externalId) => {
const id = changed.get(externalId);
return id ? [id] : [];
}),
};
},
{ timeout: 60_000 },
);
}
/**
* Keyset-paginated search, newest capture first, undated assets last. The cursor is
* the (capturedAt, id) of the previous page's last row, so inserts never shift pages.
*/
async function searchAssets(
spaceId: string,
query: MediaAssetSearch,
): Promise<MediaPage<MediaAsset>> {
const limit = pageLimit(query.limit);
const cursor = query.cursor ? decodeKeysetCursor(query.cursor) : null;
const conditions: Prisma.Sql[] = [Prisma.sql`a."spaceId" = ${spaceId}`];
if (query.sourceId) conditions.push(Prisma.sql`a."sourceId" = ${query.sourceId}`);
if (query.kinds && query.kinds.length > 0) {
conditions.push(Prisma.sql`a.kind IN (${Prisma.join(query.kinds)})`);
}
if (query.ingestStates && query.ingestStates.length > 0) {
conditions.push(Prisma.sql`a."ingestState" IN (${Prisma.join(query.ingestStates)})`);
}
if (query.from) conditions.push(Prisma.sql`a."capturedAt" >= ${new Date(query.from)}`);
if (query.to) conditions.push(Prisma.sql`a."capturedAt" <= ${new Date(query.to)}`);
if (query.favoritesOnly) conditions.push(Prisma.sql`a."isFavorite" = TRUE`);
if (query.minQuality !== undefined) {
conditions.push(Prisma.sql`a."qualityScore" >= ${query.minQuality}::float8`);
}
if (query.near) conditions.push(nearSql(query.near));
if (query.personId) {
conditions.push(Prisma.sql`EXISTS (
SELECT 1 FROM media_face_tracks ft
WHERE ft."assetId" = a.id AND ft."spaceId" = ${spaceId} AND ft."personId" = ${query.personId}
)`);
}
if (query.collectionId) {
conditions.push(Prisma.sql`EXISTS (
SELECT 1 FROM media_collection_items ci
JOIN media_collections c ON c.id = ci."collectionId"
WHERE ci."assetId" = a.id AND c.id = ${query.collectionId} AND c."spaceId" = ${spaceId}
)`);
}
const text = query.text?.trim();
if (text) conditions.push(textSql(spaceId, text));
conditions.push(keysetAfterSql(cursor, Prisma.sql`a."capturedAt"`, Prisma.sql`a.id`));
const keys = await prisma.$queryRaw<Array<{ id: string }>>(Prisma.sql`
SELECT a.id FROM media_assets a
WHERE ${Prisma.join(conditions, " AND ")}
ORDER BY a."capturedAt" DESC NULLS LAST, a.id DESC
LIMIT ${limit + 1}
`);
const rows = await prisma.mediaAsset.findMany({
where: { spaceId, id: { in: keys.map((key) => key.id) } },
});
const byId = new Map(rows.map((row) => [row.id, row]));
const ordered = keys.flatMap((key) => {
const row = byId.get(key.id);
return row ? [row] : [];
});
return toPage(ordered, limit, (row) => ({ at: row.capturedAt, id: row.id }), mapMediaAsset);
}
async function getAsset(spaceId: string, assetId: string): Promise<MediaAssetDetail> {
const asset = await requireAsset(prisma, spaceId, assetId);
const [segments, faceTracks, transcript] = await Promise.all([
prisma.mediaSegment.findMany({
where: { assetId, spaceId },
orderBy: [{ startMs: "asc" }, { id: "asc" }],
}),
prisma.mediaFaceTrack.findMany({
where: { assetId, spaceId },
orderBy: [{ startMs: "asc" }, { id: "asc" }],
}),
prisma.mediaAnalysis.findFirst({
where: { assetId, spaceId, kind: "transcript" },
orderBy: [{ createdAt: "desc" }, { id: "desc" }],
select: { data: true },
}),
]);
return {
asset: mapMediaAsset(asset),
segments: segments.map(mapMediaSegment),
faceTracks: faceTracks.map(mapMediaFaceTrack),
transcript: transcriptText(transcript?.data ?? null),
};
}
/** Full row for workers: storage path, hashes, derived file paths. */
async function getAssetRecord(spaceId: string, assetId: string): Promise<MediaAssetRow> {
return requireAsset(prisma, spaceId, assetId);
}
async function listAssetRecords(spaceId: string, assetIds: string[]): Promise<MediaAssetRow[]> {
if (assetIds.length === 0) return [];
return prisma.mediaAsset.findMany({
where: { spaceId, id: { in: [...new Set(assetIds)] } },
orderBy: [{ capturedAt: { sort: "asc", nulls: "last" } }, { id: "asc" }],
});
}
/** Every asset of the space in capture order, with the fields clustering reads. */
async function listAssetsForClustering(spaceId: string) {
return prisma.mediaAsset.findMany({
where: { spaceId },
orderBy: [{ capturedAt: { sort: "asc", nulls: "last" } }, { id: "asc" }],
select: {
id: true,
kind: true,
capturedAt: true,
durationMs: true,
gpsLat: true,
gpsLon: true,
deviceModel: true,
albumNames: true,
contentHash: true,
},
});
}
/**
* Atomic ingest state change: only applies when the asset is currently in one of
* `from`, so two workers cannot both take the same step. Returns whether it applied.
*/
async function transitionIngestState(
spaceId: string,
assetId: string,
transition: { from: MediaIngestState[]; to: MediaIngestState; error?: string | null },
): Promise<boolean> {
const result = await prisma.mediaAsset.updateMany({
where: { id: assetId, spaceId, ingestState: { in: transition.from } },
data: {
ingestState: transition.to,
ingestError: transition.to === "failed" ? (transition.error ?? null) : null,
},
});
if (result.count === 1) return true;
await requireAsset(prisma, spaceId, assetId);
return false;
}
async function recordProxy(
spaceId: string,
assetId: string,
proxy: MediaProxyResult,
): Promise<MediaAsset> {
return updateAsset(spaceId, assetId, {
contentHash: proxy.contentHash,
proxyPath: proxy.proxyPath,
thumbPath: proxy.thumbPath,
...(proxy.spritePath !== undefined ? { spritePath: proxy.spritePath } : {}),
...(proxy.durationMs !== undefined ? { durationMs: proxy.durationMs } : {}),
...(proxy.width !== undefined ? { width: proxy.width } : {}),
...(proxy.height !== undefined ? { height: proxy.height } : {}),
...(proxy.fps !== undefined ? { fps: proxy.fps } : {}),
...(proxy.codec !== undefined ? { codec: proxy.codec } : {}),
...(proxy.orientation !== undefined ? { orientation: proxy.orientation } : {}),
ingestState: "proxied",
ingestError: null,
});
}
async function recordAnalysisVersion(
spaceId: string,
assetId: string,
analysisVersion: number,
qualityScore?: number | null,
): Promise<MediaAsset> {
return updateAsset(spaceId, assetId, {
analysisVersion,
...(qualityScore !== undefined ? { qualityScore } : {}),
ingestState: "analyzed",
ingestError: null,
});
}
/** Remote assets move to awaiting_upload; returns the ids that now wait for bytes. */
async function requestUpload(spaceId: string, assetIds: string[]): Promise<string[]> {
const ids = [...new Set(assetIds)];
const owned = await prisma.mediaAsset.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
await prisma.mediaAsset.updateMany({
where: {
spaceId,
id: { in: ids },
storage: "remote",
ingestState: { in: ["indexed", "failed"] },
},
data: { ingestState: "awaiting_upload", ingestError: null },
});
const awaiting = await prisma.mediaAsset.findMany({
where: { spaceId, id: { in: ids }, ingestState: "awaiting_upload" },
select: { id: true },
});
const set = new Set(awaiting.map((row) => row.id));
return ids.filter((id) => set.has(id));
}
async function assetsNeedingWork(
spaceId: string,
state: MediaIngestState,
limit: number,
): Promise<MediaAssetRow[]> {
return prisma.mediaAsset.findMany({
where: { spaceId, ingestState: state },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
take: pageLimit(limit),
});
}
// -------------------------------------------------------------------------
// Analyses
// -------------------------------------------------------------------------
/** Insert or replace the result for (asset, kind, modelVersion); other versions stay. */
async function upsertAnalysis(
spaceId: string,
assetId: string,
analysis: NewMediaAnalysis,
): Promise<MediaAnalysisRow> {
await requireAsset(prisma, spaceId, assetId);
const data = {
modelId: analysis.modelId,
data: analysis.data ?? Prisma.DbNull,
dataPath: analysis.dataPath ?? null,
vectors: analysis.vectors ? toBytes(analysis.vectors) : null,
};
return prisma.mediaAnalysis.upsert({
where: {
assetId_kind_modelVersion: {
assetId,
kind: analysis.kind,
modelVersion: analysis.modelVersion,
},
},
create: {
...data,
spaceId,
assetId,
kind: analysis.kind,
modelVersion: analysis.modelVersion,
},
update: data,
});
}
async function latestAnalysis(
spaceId: string,
assetId: string,
kind: MediaAnalysisKind,
): Promise<MediaAnalysisRow | null> {
await requireAsset(prisma, spaceId, assetId);
return prisma.mediaAnalysis.findFirst({
where: { spaceId, assetId, kind },
orderBy: [{ createdAt: "desc" }, { id: "desc" }],
});
}
/** Latest analysis of one kind per asset across the space (e.g. embeddings for clustering). */
async function latestAnalysesByKind(
spaceId: string,
kind: MediaAnalysisKind,
): Promise<Map<string, MediaAnalysisRow>> {
const rows = await prisma.mediaAnalysis.findMany({
where: { spaceId, kind },
orderBy: [{ assetId: "asc" }, { createdAt: "desc" }, { id: "desc" }],
});
const latest = new Map<string, MediaAnalysisRow>();
for (const row of rows) if (!latest.has(row.assetId)) latest.set(row.assetId, row);
return latest;
}
// -------------------------------------------------------------------------
// Segments
// -------------------------------------------------------------------------
/** Replace every segment of the asset with the result of one analysis version. */
async function replaceSegments(
spaceId: string,
assetId: string,
analysisVersion: number,
segments: NewMediaSegment[],
): Promise<number> {
for (const segment of segments) {
if (segment.endMs <= segment.startMs || segment.startMs < 0) {
throw new InvalidMediaInputError("Segment range is empty");
}
}
return prisma.$transaction(async (tx) => {
await requireAsset(tx, spaceId, assetId);
await lockMediaKey(tx, `segments:${assetId}`);
await tx.mediaSegment.deleteMany({ where: { assetId, spaceId } });
const created = await tx.mediaSegment.createMany({
data: segments.map((segment) => ({
spaceId,
assetId,
analysisVersion,
kind: segment.kind,
startMs: segment.startMs,
endMs: segment.endMs,
scores: segment.scores ?? {},
teaserScore: segment.teaserScore ?? null,
transcriptExcerpt: segment.transcriptExcerpt ?? null,
labels: segment.labels ?? [],
paidOnly: segment.paidOnly ?? false,
})),
});
return created.count;
});
}
async function listSegments(spaceId: string, assetIds: string[]): Promise<MediaSegment[]> {
if (assetIds.length === 0) return [];
const rows = await prisma.mediaSegment.findMany({
where: { spaceId, assetId: { in: [...new Set(assetIds)] } },
orderBy: [{ assetId: "asc" }, { startMs: "asc" }, { id: "asc" }],
});
return rows.map(mapMediaSegment);
}
/** All or nothing: any id outside the space fails the whole call. */
async function setSegmentsPaidOnly(
spaceId: string,
segmentIds: string[],
paidOnly: boolean,
): Promise<number> {
const ids = [...new Set(segmentIds)];
return prisma.$transaction(async (tx) => {
const result = await tx.mediaSegment.updateMany({
where: { spaceId, id: { in: ids } },
data: { paidOnly },
});
if (result.count !== ids.length) throw new IsolationError();
return result.count;
});
}
return {
createSource,
listSources,
getSource,
updateSourceSync,
removeSource,
removeSourceItems,
mediaSpaceOf,
upsertSourceItems,
searchAssets,
getAsset,
getAssetRecord,
listAssetRecords,
listAssetsForClustering,
transitionIngestState,
recordProxy,
recordAnalysisVersion,
requestUpload,
assetsNeedingWork,
upsertAnalysis,
latestAnalysis,
latestAnalysesByKind,
replaceSegments,
listSegments,
setSegmentsPaidOnly,
};
}

View File

@ -0,0 +1,598 @@
import type {
AnonymizeMode,
ConsentInput,
MarkTreatment,
MediaFaceTrack,
MediaMark,
MediaMarkKind,
MediaMarkPrompt,
MediaPerson,
} from "@rakazo/contracts";
import type { MediaMark as MediaMarkRow, PrismaClient } from "./client.js";
import type { MediaDb } from "./media-shared.js";
import {
assertAssetsInSpace,
InvalidMediaInputError,
lockMediaKey,
mapMediaFaceTrack,
mapMediaMark,
mapMediaPerson,
toBytes,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
export interface MediaPersonSuggestion {
/** Float32 mean identity embedding. Biometric: stays in this database. */
centroid: Uint8Array;
trackIds: string[];
}
export interface MediaPersonPatch {
displayName?: string;
isSelf?: boolean;
confirmed?: boolean;
anonymizeByDefault?: AnonymizeMode;
syntheticSeed?: number | null;
}
/**
* Who records consent. A user id is mandatory by type and checked against the space,
* so a caller without an authenticated user (an agent tool) cannot produce one.
*/
export interface ConsentRecorder {
spaceId: string;
recordedByUserId: string;
}
export interface RemovedPerson {
deletedFaceTracks: number;
deletedMarks: number;
/** Files the caller must delete too: box keyframes and mark masks. */
keyframesPaths: string[];
maskDirs: string[];
}
export interface NewMediaFaceTrack {
startMs: number;
endMs: number;
confidenceMin: number;
keyframesPath: string;
/** Float32 identity embedding. Biometric: stays in this database. */
embedding: Uint8Array | null;
personId?: string | null;
}
export interface MediaFaceTrackRecord extends MediaFaceTrack {
keyframesPath: string;
modelVersion: string;
}
export interface MediaFaceEmbedding {
trackId: string;
assetId: string;
personId: string | null;
embedding: Uint8Array;
}
export interface NewMediaMark {
kind: MediaMarkKind;
label: string | null;
personId: string | null;
assetId: string;
prompt: MediaMarkPrompt;
treatment: MarkTreatment;
}
export interface MediaMarkPatch {
treatment?: MarkTreatment;
label?: string | null;
personId?: string | null;
}
export interface MediaMarkResult {
qualityScore: number | null;
suggestedFallback: MarkTreatment | null;
maskDir: string | null;
}
export interface MediaMarkRecord extends MediaMark {
maskDir: string | null;
}
function markRecord(row: MediaMarkRow): MediaMarkRecord {
return { ...mapMediaMark(row), maskDir: row.maskDir };
}
/** People, face tracks and identifying marks. Biometric data stays in this database. */
export function createMediaPeopleRepos(prisma: PrismaClient) {
// -------------------------------------------------------------------------
// People
// -------------------------------------------------------------------------
const personInclude = {
_count: { select: { faceTracks: true } },
faceTracks: {
orderBy: [{ confidenceMin: "desc" as const }, { id: "asc" as const }],
take: 1,
select: { assetId: true },
},
};
async function personDto(db: MediaDb, spaceId: string, personId: string): Promise<MediaPerson> {
const row = await db.mediaPerson.findFirst({
where: { id: personId, spaceId },
include: personInclude,
});
if (!row) throw new IsolationError();
return mapMediaPerson(row, row._count.faceTracks, row.faceTracks[0]?.assetId ?? null);
}
async function listPeople(spaceId: string): Promise<MediaPerson[]> {
const rows = await prisma.mediaPerson.findMany({
where: { spaceId },
orderBy: [{ isSelf: "desc" }, { createdAt: "asc" }, { id: "asc" }],
include: personInclude,
});
return rows.map((row) =>
mapMediaPerson(row, row._count.faceTracks, row.faceTracks[0]?.assetId ?? null),
);
}
async function getPerson(spaceId: string, personId: string): Promise<MediaPerson> {
return personDto(prisma, spaceId, personId);
}
/** People with consent fields and seed, for the render gate. */
async function listPersonRecords(spaceId: string, personIds?: string[]) {
return prisma.mediaPerson.findMany({
where: { spaceId, ...(personIds ? { id: { in: personIds } } : {}) },
omit: { centroid: true },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
});
}
/** Identity centroids for matching new face tracks to known people. Local only. */
async function personCentroids(
spaceId: string,
): Promise<Array<{ personId: string; centroid: Uint8Array }>> {
const rows = await prisma.mediaPerson.findMany({
where: { spaceId, centroid: { not: null } },
select: { id: true, centroid: true },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
});
return rows.map((row) => ({ personId: row.id, centroid: row.centroid as Uint8Array }));
}
/**
* Turn face clusters into unconfirmed people ("Person N"). Only tracks that have no
* person yet are assigned, so a user's earlier assignment is never overwritten.
*/
async function createPersonSuggestions(
spaceId: string,
suggestions: MediaPersonSuggestion[],
): Promise<MediaPerson[]> {
if (suggestions.length === 0) return [];
const ids = await prisma.$transaction(async (tx) => {
await lockMediaKey(tx, `people:${spaceId}`);
const trackIds = [...new Set(suggestions.flatMap((suggestion) => suggestion.trackIds))];
const owned = await tx.mediaFaceTrack.count({ where: { spaceId, id: { in: trackIds } } });
if (owned !== trackIds.length) throw new IsolationError();
let number = await tx.mediaPerson.count({ where: { spaceId } });
const created: string[] = [];
for (const suggestion of suggestions) {
number += 1;
const person = await tx.mediaPerson.create({
data: {
spaceId,
displayName: `Person ${number}`,
centroid: toBytes(suggestion.centroid),
},
select: { id: true },
});
await tx.mediaFaceTrack.updateMany({
where: { spaceId, id: { in: suggestion.trackIds }, personId: null },
data: { personId: person.id },
});
created.push(person.id);
}
return created;
});
return Promise.all(ids.map((id) => personDto(prisma, spaceId, id)));
}
async function updatePerson(
spaceId: string,
personId: string,
patch: MediaPersonPatch,
): Promise<MediaPerson> {
await prisma.$transaction(async (tx) => {
const result = await tx.mediaPerson.updateMany({
where: { id: personId, spaceId },
data: {
...(patch.displayName !== undefined ? { displayName: patch.displayName } : {}),
...(patch.isSelf !== undefined ? { isSelf: patch.isSelf } : {}),
...(patch.confirmed !== undefined ? { confirmed: patch.confirmed } : {}),
...(patch.anonymizeByDefault !== undefined
? { anonymizeByDefault: patch.anonymizeByDefault }
: {}),
...(patch.syntheticSeed !== undefined ? { syntheticSeed: patch.syntheticSeed } : {}),
},
});
if (result.count !== 1) throw new IsolationError();
// One person per space is the user themself.
if (patch.isSelf) {
await tx.mediaPerson.updateMany({
where: { spaceId, isSelf: true, id: { not: personId } },
data: { isSelf: false },
});
}
});
return personDto(prisma, spaceId, personId);
}
/**
* Merge into the first person: tracks and marks move, the others are deleted along
* with their centroids. The target keeps its own consent record.
*/
async function mergePeople(spaceId: string, personIds: string[]): Promise<MediaPerson> {
const ids = [...new Set(personIds)];
if (ids.length < 2) throw new InvalidMediaInputError("Merge needs two people");
const [targetId, ...others] = ids as [string, ...string[]];
await prisma.$transaction(async (tx) => {
const owned = await tx.mediaPerson.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
await tx.mediaFaceTrack.updateMany({
where: { spaceId, personId: { in: others } },
data: { personId: targetId },
});
await tx.mediaMark.updateMany({
where: { spaceId, personId: { in: others } },
data: { personId: targetId },
});
await tx.mediaPerson.updateMany({
where: { spaceId, id: { in: others } },
data: { centroid: null },
});
await tx.mediaPerson.deleteMany({ where: { spaceId, id: { in: others } } });
await tx.mediaPerson.update({ where: { id: targetId }, data: { confirmed: true } });
});
return personDto(prisma, spaceId, targetId);
}
async function assignTrack(
spaceId: string,
trackId: string,
personId: string | null,
): Promise<MediaFaceTrack> {
return prisma.$transaction(async (tx) => {
if (personId) {
const person = await tx.mediaPerson.findFirst({
where: { id: personId, spaceId },
select: { id: true },
});
if (!person) throw new IsolationError();
}
const result = await tx.mediaFaceTrack.updateMany({
where: { id: trackId, spaceId },
data: { personId },
});
if (result.count !== 1) throw new IsolationError();
const row = await tx.mediaFaceTrack.findUniqueOrThrow({ where: { id: trackId } });
return mapMediaFaceTrack(row);
});
}
/**
* Record consent and age verification for a person. Only a signed-in member of the
* space can do this; the recorder is stored with the record.
*/
async function recordConsent(
recorder: ConsentRecorder,
input: ConsentInput,
): Promise<MediaPerson> {
const { spaceId, recordedByUserId } = recorder;
if (typeof recordedByUserId !== "string" || recordedByUserId.trim() === "") {
throw new InvalidMediaInputError("Consent needs the recording user");
}
if (input.ageVerificationMethod !== "none" && !input.ageVerifiedAt) {
throw new InvalidMediaInputError("Age verification needs a date");
}
const member = await prisma.spaceMember.findFirst({
where: { spaceId, userId: recordedByUserId },
select: { id: true },
});
if (!member) throw new IsolationError();
const result = await prisma.mediaPerson.updateMany({
where: { id: input.personId, spaceId },
data: {
consentArtifactId: input.consentArtifactId,
consentRecordedAt: new Date(input.consentRecordedAt),
consentRecordedBy: recordedByUserId,
ageVerificationMethod: input.ageVerificationMethod,
ageVerifiedAt:
input.ageVerificationMethod === "none" || !input.ageVerifiedAt
? null
: new Date(input.ageVerifiedAt),
},
});
if (result.count !== 1) throw new IsolationError();
return personDto(prisma, spaceId, input.personId);
}
/**
* Delete a person and every biometric record tied to them: the centroid, the
* embeddings of their face tracks, the tracks themselves, and person-bound marks.
* Embeddings are blanked before the rows go so no copy survives in the same
* transaction's row versions; the caller deletes the returned files.
*/
async function removePersonCompletely(spaceId: string, personId: string): Promise<RemovedPerson> {
return prisma.$transaction(async (tx) => {
const person = await tx.mediaPerson.findFirst({
where: { id: personId, spaceId },
select: { id: true },
});
if (!person) throw new IsolationError();
const tracks = await tx.mediaFaceTrack.findMany({
where: { spaceId, personId },
select: { keyframesPath: true },
});
const marks = await tx.mediaMark.findMany({
where: { spaceId, personId },
select: { maskDir: true },
});
await tx.mediaFaceTrack.updateMany({
where: { spaceId, personId },
data: { embedding: null },
});
const deletedTracks = await tx.mediaFaceTrack.deleteMany({ where: { spaceId, personId } });
const deletedMarks = await tx.mediaMark.deleteMany({ where: { spaceId, personId } });
await tx.mediaPerson.update({ where: { id: personId }, data: { centroid: null } });
await tx.mediaPerson.delete({ where: { id: personId } });
return {
deletedFaceTracks: deletedTracks.count,
deletedMarks: deletedMarks.count,
keyframesPaths: tracks.map((track) => track.keyframesPath),
maskDirs: marks.flatMap((mark) => (mark.maskDir ? [mark.maskDir] : [])),
};
});
}
// -------------------------------------------------------------------------
// Face tracks
// -------------------------------------------------------------------------
/** Replace the face tracks of one asset with a fresh tracking run. */
async function replaceFaceTracks(
spaceId: string,
assetId: string,
modelVersion: string,
tracks: NewMediaFaceTrack[],
): Promise<MediaFaceTrack[]> {
return prisma.$transaction(async (tx) => {
const asset = await tx.mediaAsset.findFirst({
where: { id: assetId, spaceId },
select: { id: true },
});
if (!asset) throw new IsolationError();
const personIds = [
...new Set(tracks.flatMap((track) => (track.personId ? [track.personId] : []))),
];
if (personIds.length > 0) {
const owned = await tx.mediaPerson.count({ where: { spaceId, id: { in: personIds } } });
if (owned !== personIds.length) throw new IsolationError();
}
await lockMediaKey(tx, `faces:${assetId}`);
await tx.mediaFaceTrack.updateMany({
where: { spaceId, assetId },
data: { embedding: null },
});
await tx.mediaFaceTrack.deleteMany({ where: { spaceId, assetId } });
const rows = await tx.mediaFaceTrack.createManyAndReturn({
data: tracks.map((track) => ({
spaceId,
assetId,
modelVersion,
startMs: track.startMs,
endMs: track.endMs,
confidenceMin: track.confidenceMin,
keyframesPath: track.keyframesPath,
embedding: track.embedding ? toBytes(track.embedding) : null,
personId: track.personId ?? null,
})),
omit: { embedding: true },
});
return rows
.sort((a, b) => a.startMs - b.startMs || a.id.localeCompare(b.id))
.map(mapMediaFaceTrack);
});
}
async function listFaceTracks(
spaceId: string,
filter: { assetId?: string; personId?: string } = {},
): Promise<MediaFaceTrack[]> {
const rows = await prisma.mediaFaceTrack.findMany({
where: { spaceId, ...filter },
omit: { embedding: true },
orderBy: [{ assetId: "asc" }, { startMs: "asc" }, { id: "asc" }],
});
return rows.map(mapMediaFaceTrack);
}
/** Tracks with their keyframe files, for anonymization and the gate. */
async function faceTrackRecords(
spaceId: string,
assetIds: string[],
): Promise<MediaFaceTrackRecord[]> {
if (assetIds.length === 0) return [];
const rows = await prisma.mediaFaceTrack.findMany({
where: { spaceId, assetId: { in: [...new Set(assetIds)] } },
omit: { embedding: true },
orderBy: [{ assetId: "asc" }, { startMs: "asc" }, { id: "asc" }],
});
return rows.map((row) => ({
...mapMediaFaceTrack(row),
keyframesPath: row.keyframesPath,
modelVersion: row.modelVersion,
}));
}
/** Identity embeddings for local person clustering. Never leaves the host. */
async function faceTrackEmbeddings(
spaceId: string,
options: { unassignedOnly?: boolean } = {},
): Promise<MediaFaceEmbedding[]> {
const rows = await prisma.mediaFaceTrack.findMany({
where: {
spaceId,
embedding: { not: null },
...(options.unassignedOnly ? { personId: null } : {}),
},
select: { id: true, assetId: true, personId: true, embedding: true },
orderBy: [{ assetId: "asc" }, { startMs: "asc" }, { id: "asc" }],
});
return rows.map((row) => ({
trackId: row.id,
assetId: row.assetId,
personId: row.personId,
embedding: row.embedding as Uint8Array,
}));
}
// -------------------------------------------------------------------------
// Marks
// -------------------------------------------------------------------------
async function assertPerson(
db: MediaDb,
spaceId: string,
personId: string | null,
): Promise<void> {
if (!personId) return;
const person = await db.mediaPerson.findFirst({
where: { id: personId, spaceId },
select: { id: true },
});
if (!person) throw new IsolationError();
}
async function requireMark(spaceId: string, markId: string): Promise<MediaMarkRow> {
const row = await prisma.mediaMark.findFirst({ where: { id: markId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
async function createMark(spaceId: string, input: NewMediaMark): Promise<MediaMark> {
await assertAssetsInSpace(prisma, spaceId, [input.assetId]);
await assertPerson(prisma, spaceId, input.personId);
const row = await prisma.mediaMark.create({
data: {
spaceId,
kind: input.kind,
label: input.label,
personId: input.personId,
assetId: input.assetId,
prompt: input.prompt,
treatment: input.treatment,
},
});
return mapMediaMark(row);
}
async function listMarks(
spaceId: string,
filter: { personId?: string; assetId?: string } = {},
): Promise<MediaMark[]> {
const rows = await prisma.mediaMark.findMany({
where: { spaceId, ...filter },
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
});
return rows.map(mapMediaMark);
}
/**
* Marks that apply to an asset: those drawn on it plus the standing marks of every
* person seen in it (person rules hold across all projects).
*/
async function marksForAsset(spaceId: string, assetId: string): Promise<MediaMarkRecord[]> {
const rows = await prisma.mediaMark.findMany({
where: {
spaceId,
OR: [{ assetId }, { person: { faceTracks: { some: { assetId, spaceId } } } }],
},
orderBy: [{ createdAt: "asc" }, { id: "asc" }],
});
return rows.map(markRecord);
}
async function getMark(spaceId: string, markId: string): Promise<MediaMarkRecord> {
return markRecord(await requireMark(spaceId, markId));
}
async function updateMark(
spaceId: string,
markId: string,
patch: MediaMarkPatch,
): Promise<MediaMark> {
if (patch.personId !== undefined) await assertPerson(prisma, spaceId, patch.personId);
const result = await prisma.mediaMark.updateMany({
where: { id: markId, spaceId },
data: {
...(patch.treatment !== undefined ? { treatment: patch.treatment } : {}),
...(patch.label !== undefined ? { label: patch.label } : {}),
...(patch.personId !== undefined ? { personId: patch.personId } : {}),
},
});
if (result.count !== 1) throw new IsolationError();
return mapMediaMark(await requireMark(spaceId, markId));
}
/** Store the segmentation/treatment outcome: mask, quality, and any fallback proposal. */
async function recordMarkResult(
spaceId: string,
markId: string,
outcome: MediaMarkResult,
): Promise<MediaMarkRecord> {
const result = await prisma.mediaMark.updateMany({
where: { id: markId, spaceId },
data: {
qualityScore: outcome.qualityScore,
suggestedFallback: outcome.suggestedFallback,
maskDir: outcome.maskDir,
},
});
if (result.count !== 1) throw new IsolationError();
return getMark(spaceId, markId);
}
async function removeMark(spaceId: string, markId: string): Promise<{ maskDir: string | null }> {
const row = await requireMark(spaceId, markId);
const result = await prisma.mediaMark.deleteMany({ where: { id: markId, spaceId } });
if (result.count !== 1) throw new IsolationError();
return { maskDir: row.maskDir };
}
return {
listPeople,
getPerson,
listPersonRecords,
personCentroids,
createPersonSuggestions,
updatePerson,
mergePeople,
assignTrack,
recordConsent,
removePersonCompletely,
replaceFaceTracks,
listFaceTracks,
faceTrackRecords,
faceTrackEmbeddings,
createMark,
listMarks,
marksForAsset,
getMark,
updateMark,
recordMarkResult,
removeMark,
};
}

View File

@ -0,0 +1,462 @@
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import type { MediaRepos, NewMediaPublication } from "./media.js";
import { createMediaRepos } from "./media.js";
import type { MediaTestDb } from "./media-fixtures.js";
import {
BASE_TIME,
openMediaTestDb,
sourceItem,
testBrandKit,
testEdl,
testTeaserWeights,
} from "./media-fixtures.js";
import { InvalidMediaInputError, MediaStateConflictError } from "./media-shared.js";
import { IsolationError } from "./scope.js";
const databaseUrl = process.env.DATABASE_URL;
const describePostgres =
process.env.VERIFY_DATABASE && databaseUrl ? describe.sequential : describe.skip;
const PROFILE = "example-short-vertical";
describePostgres("media production, publishing and metrics (PostgreSQL)", () => {
let db: MediaTestDb;
let media: MediaRepos;
beforeAll(() => {
db = openMediaTestDb(databaseUrl!);
media = createMediaRepos(db.prisma);
});
afterAll(async () => {
await db?.cleanup();
});
async function newProject() {
const { spaceId, userId } = await db.newUser();
const source = await media.createSource(spaceId, {
kind: "emulator",
label: "Emulator",
ownerUserId: userId,
config: {},
});
const { changedAssetIds } = await media.upsertSourceItems(spaceId, source.id, [sourceItem(1)]);
const assetId = changedAssetIds[0] as string;
const project = await media.createProject(spaceId, userId, {
title: "Launch",
goal: "teaser",
profileIds: [PROFILE, "example-paid-video"],
assetIds: [assetId],
});
return { spaceId, userId, assetId, projectId: project.id };
}
async function doneRender(spaceId: string, userId: string, projectId: string, assetId: string) {
const plan = await media.createPlanVersion(spaceId, {
projectId,
profileId: PROFILE,
edl: testEdl(assetId),
createdBy: "user",
rationale: null,
});
const render = await media.createRender(spaceId, {
planId: plan.id,
quality: "final",
createdBy: userId,
});
await media.claimRender(spaceId, render.id);
await media.updateRender(spaceId, render.id, { status: "done", progress: 1 });
return render.id;
}
const publicationInput = (renderId: string, content: string): NewMediaPublication => ({
renderId,
destination: "bundle",
targetUrl: "https://example.com/offer",
trackingUrl: `https://example.com/offer?utm_content=${content}`,
utm: { source: "shorts", medium: "video", campaign: "launch", content },
title: null,
caption: null,
tags: [],
thumbnailVariant: `thumb-${content}`,
scheduledAt: null,
});
it("validates and stores brand kits", async () => {
const { spaceId } = await db.newUser();
await expect(
media.upsertBrandKit(spaceId, { kit: { ...testBrandKit, colors: { primary: "red" } } }),
).rejects.toBeInstanceOf(InvalidMediaInputError);
const kit = await media.upsertBrandKit(spaceId, { kit: testBrandKit });
const renamed = await media.upsertBrandKit(spaceId, {
id: kit.id,
kit: { ...testBrandKit, name: "Renamed" },
});
expect(renamed).toMatchObject({ id: kit.id, kit: { name: "Renamed" } });
expect(await media.listBrandKits(spaceId)).toHaveLength(1);
await media.removeBrandKit(spaceId, kit.id);
expect(await media.listBrandKits(spaceId)).toEqual([]);
});
it("creates, pages, updates and removes projects", async () => {
const { spaceId, userId } = await db.newUser();
await expect(
media.createProject(spaceId, userId, {
title: "Bad",
goal: "teaser",
profileIds: [PROFILE],
assetIds: ["missing"],
}),
).rejects.toBeInstanceOf(IsolationError);
const created = [];
for (let index = 0; index < 5; index++) {
created.push(
await media.createProject(spaceId, userId, {
title: `Project ${index}`,
goal: "highlight",
profileIds: [PROFILE, PROFILE],
}),
);
}
expect(created[0]?.profileIds).toEqual([PROFILE]);
const seen: string[] = [];
let cursor: string | null = null;
do {
const page = await media.listProjects(spaceId, { cursor, limit: 2 });
seen.push(...page.items.map((project) => project.title));
cursor = page.nextCursor;
} while (cursor);
expect(seen).toEqual(["Project 4", "Project 3", "Project 2", "Project 1", "Project 0"]);
const updated = await media.updateProject(spaceId, created[0]?.id as string, {
title: "Renamed",
status: "review",
});
expect(updated).toMatchObject({ title: "Renamed", status: "review" });
expect((await media.listProjects(spaceId, { limit: 1 })).items[0]?.id).toBe(created[0]?.id);
await media.removeProject(spaceId, created[0]?.id as string);
await expect(media.getProject(spaceId, created[0]?.id as string)).rejects.toBeInstanceOf(
IsolationError,
);
});
it("numbers concurrent plan versions without gaps or duplicates", async () => {
const { spaceId, assetId, projectId } = await newProject();
await expect(
media.createPlanVersion(spaceId, {
projectId,
profileId: PROFILE,
edl: { edlVersion: 1 },
createdBy: "agent",
rationale: null,
}),
).rejects.toBeInstanceOf(InvalidMediaInputError);
await expect(
media.createPlanVersion(spaceId, {
projectId,
profileId: "example-paid-video",
edl: testEdl(assetId),
createdBy: "agent",
rationale: null,
}),
).rejects.toBeInstanceOf(InvalidMediaInputError);
const plans = await Promise.all(
Array.from({ length: 12 }, (_, index) =>
media.createPlanVersion(spaceId, {
projectId,
profileId: PROFILE,
edl: testEdl(assetId),
createdBy: index % 2 ? "agent" : "user",
rationale: `try ${index}`,
}),
),
);
expect(plans.map((plan) => plan.version).sort((a, b) => a - b)).toEqual(
Array.from({ length: 12 }, (_, index) => index + 1),
);
await media.createPlanVersion(spaceId, {
projectId,
profileId: "example-paid-video",
edl: testEdl(assetId, "example-paid-video"),
createdBy: "agent",
rationale: null,
});
const latest = await media.latestPlans(spaceId, projectId);
expect(latest.map((plan) => [plan.profileId, plan.version])).toEqual([
["example-paid-video", 1],
[PROFILE, 12],
]);
expect((await media.latestPlan(spaceId, projectId, PROFILE))?.version).toBe(12);
expect(await media.listPlanVersions(spaceId, projectId, PROFILE)).toHaveLength(12);
expect(latest[1]?.edl.tracks.video[0]?.speed).toBe(1);
});
it("lets exactly one worker claim a render and never revives a canceled one", async () => {
const { spaceId, userId, assetId, projectId } = await newProject();
const plan = await media.createPlanVersion(spaceId, {
projectId,
profileId: PROFILE,
edl: testEdl(assetId),
createdBy: "agent",
rationale: null,
});
const render = await media.createRender(spaceId, {
planId: plan.id,
quality: "draft",
createdBy: userId,
});
expect(render).toMatchObject({ status: "queued", projectId, profileId: PROFILE, progress: 0 });
const claims = await Promise.all(
Array.from({ length: 5 }, () => media.claimRender(spaceId, render.id)),
);
expect(claims.filter((claim) => claim !== null)).toHaveLength(1);
const progressed = await media.updateRender(spaceId, render.id, { progress: 0.4 });
expect(progressed?.progress).toBe(0.4);
await expect(
media.updateRender(spaceId, render.id, { autoFixRound: 4 }),
).rejects.toBeInstanceOf(InvalidMediaInputError);
const canceled = await media.cancelRender(spaceId, render.id);
expect(canceled.status).toBe("canceled");
expect(await media.updateRender(spaceId, render.id, { status: "done" })).toBeNull();
expect((await media.getRender(spaceId, render.id)).status).toBe("canceled");
await expect(media.cancelRender(spaceId, render.id)).rejects.toBeInstanceOf(
MediaStateConflictError,
);
const second = await media.createRender(spaceId, {
planId: plan.id,
quality: "final",
createdBy: userId,
});
await media.claimRender(spaceId, second.id);
const gated = await media.updateRender(spaceId, second.id, {
status: "blocked",
progress: 1,
autoFixRound: 3,
outputs: [
{
kind: "video",
artifactId: null,
fileName: "out.mp4",
sizeBytes: 1234,
path: "renders/x/out.mp4",
},
],
gateReport: {
profileId: PROFILE,
passed: false,
checkedAt: new Date(BASE_TIME).toISOString(),
sampledFrames: 10,
issues: [{ check: "consent", severity: "block", code: "missing_consent", message: "x" }],
autoFixRound: 3,
},
ffprobe: {
durationMs: 5000,
width: 1080,
height: 1920,
fps: 30,
videoCodec: "h264",
audioCodec: "aac",
sizeBytes: 1234,
integratedLufs: -14,
truePeakDb: -1,
tags: {},
},
});
expect(gated).toMatchObject({
status: "blocked",
outputs: [{ kind: "video", fileName: "out.mp4", sizeBytes: 1234, artifactId: null }],
gateReport: { passed: false },
ffprobe: { width: 1080 },
});
expect(gated?.outputs[0]).not.toHaveProperty("path");
const record = await media.getRenderRecord(spaceId, second.id);
expect(record.outputs).toEqual([expect.objectContaining({ path: "renders/x/out.mp4" })]);
expect(record.autoFixRound).toBe(3);
await expect(media.cancelRender(spaceId, second.id)).rejects.toBeInstanceOf(
MediaStateConflictError,
);
expect((await media.listRenders(spaceId, projectId)).map((r) => r.id)).toEqual([
second.id,
render.id,
]);
const detail = await media.getProject(spaceId, projectId);
expect(detail.plans).toHaveLength(1);
expect(detail.renders).toHaveLength(2);
});
it("publishes only finished renders and moves publications through their states", async () => {
const { spaceId, userId, assetId, projectId } = await newProject();
const plan = await media.createPlanVersion(spaceId, {
projectId,
profileId: PROFILE,
edl: testEdl(assetId),
createdBy: "agent",
rationale: null,
});
const queued = await media.createRender(spaceId, {
planId: plan.id,
quality: "final",
createdBy: userId,
});
await expect(
media.createPublication(spaceId, userId, publicationInput(queued.id, "a")),
).rejects.toBeInstanceOf(MediaStateConflictError);
const renderId = await doneRender(spaceId, userId, projectId, assetId);
const draft = await media.createPublication(spaceId, userId, publicationInput(renderId, "a"));
expect(draft).toMatchObject({ status: "draft", profileId: PROFILE, utm: { content: "a" } });
const scheduledAt = new Date(BASE_TIME + 86_400_000);
const scheduled = await media.schedulePublication(spaceId, draft.id, scheduledAt);
expect(scheduled).toMatchObject({
status: "scheduled",
scheduledAt: scheduledAt.toISOString(),
});
const bundled = await media.markPublicationBundled(spaceId, draft.id, "artifact-bundle");
expect(bundled).toMatchObject({ status: "bundled", bundleArtifactId: "artifact-bundle" });
const published = await media.markPublicationPublished(
spaceId,
draft.id,
"https://video.example.com/v/1",
);
expect(published.status).toBe("published");
expect(published.publishedAt).not.toBeNull();
await expect(media.cancelPublication(spaceId, draft.id)).rejects.toBeInstanceOf(
MediaStateConflictError,
);
const other = await media.createPublication(spaceId, userId, {
...publicationInput(renderId, "b"),
scheduledAt,
});
expect(other.status).toBe("scheduled");
expect((await media.markPublicationFailed(spaceId, other.id)).status).toBe("failed");
expect((await media.cancelPublication(spaceId, other.id)).status).toBe("canceled");
const page1 = await media.listPublications(spaceId, { projectId, limit: 1 });
expect(page1.items.map((item) => item.id)).toEqual([other.id]);
const page2 = await media.listPublications(spaceId, { projectId, cursor: page1.nextCursor });
expect(page2.items.map((item) => item.id)).toEqual([draft.id]);
expect(page2.nextCursor).toBeNull();
const elsewhere = await newProject();
expect(
(await media.listPublications(spaceId, { projectId: elsewhere.projectId })).items,
).toEqual([]);
const byContent = await media.resolvePublicationKeys(spaceId, "utm_content", ["a", "b", "z"]);
expect(Object.fromEntries(byContent)).toEqual({ a: draft.id, b: other.id });
const byUrl = await media.resolvePublicationKeys(spaceId, "external_url", [
"https://video.example.com/v/1",
]);
expect(byUrl.get("https://video.example.com/v/1")).toBe(draft.id);
const byTracking = await media.resolvePublicationKeys(spaceId, "tracking_url", [
"https://example.com/offer?utm_content=b",
]);
expect([...byTracking.values()]).toEqual([other.id]);
});
it("counts the funnel from the latest observation per source without double counting", async () => {
const { spaceId, userId, assetId, projectId } = await newProject();
const renderId = await doneRender(spaceId, userId, projectId, assetId);
const a = await media.createPublication(spaceId, userId, publicationInput(renderId, "a"));
const b = await media.createPublication(spaceId, userId, publicationInput(renderId, "b"));
const day = (n: number) => new Date(BASE_TIME + n * 86_400_000).toISOString();
const report = [
{
publicationId: a.id,
observedAt: day(1),
views: 100,
clicks: 10,
conversions: 1,
revenueCents: 500,
},
{
publicationId: a.id,
observedAt: day(2),
views: 250,
clicks: 30,
conversions: 3,
revenueCents: 1500,
},
{
publicationId: b.id,
observedAt: day(2),
views: 40,
clicks: 4,
conversions: 0,
revenueCents: 0,
},
].map((row) => ({ ...row, currency: "EUR", source: "csv_import" as const }));
expect(await media.recordMetrics(spaceId, report)).toBe(3);
expect(await media.recordMetrics(spaceId, report)).toBe(3);
await media.recordMetric(spaceId, {
publicationId: a.id,
observedAt: day(1),
views: 0,
clicks: 0,
conversions: 2,
revenueCents: 1000,
currency: "EUR",
source: "manual",
});
expect(await media.listMetrics(spaceId, a.id)).toHaveLength(3);
const rows = await media.funnel(spaceId, projectId);
const byId = new Map(rows.map((row) => [row.publicationId, row]));
expect(byId.get(a.id)).toEqual({
publicationId: a.id,
projectId,
profileId: PROFILE,
variant: "a",
thumbnailVariant: "thumb-a",
views: 250,
clicks: 30,
conversions: 5,
revenueCents: 2500,
currency: "EUR",
});
expect(byId.get(b.id)).toMatchObject({ views: 40, clicks: 4, conversions: 0 });
expect(await media.funnel(spaceId)).toHaveLength(2);
expect(await media.publicationsWithMetricsCount(spaceId)).toBe(2);
await expect(media.recordMetric(spaceId, { ...report[0]!, views: -1 })).rejects.toBeInstanceOf(
InvalidMediaInputError,
);
});
it("saves CSV mappings and the teaser model", async () => {
const { spaceId } = await db.newUser();
const mapping = {
name: "Platform export",
delimiter: ";" as const,
columns: { publication: "Link", observedAt: "Date", views: "Views", revenue: "Revenue" },
publicationKey: "tracking_url" as const,
revenueUnit: "major" as const,
defaultCurrency: "USD",
};
const saved = await media.saveMetricMapping(spaceId, mapping);
expect(saved).toMatchObject({ ...mapping, id: expect.any(String) });
const again = await media.saveMetricMapping(spaceId, { ...mapping, delimiter: "," });
expect(again.id).toBe(saved.id);
const other = await media.saveMetricMapping(spaceId, { ...mapping, name: "Other" });
await expect(
media.saveMetricMapping(spaceId, { ...mapping, id: other.id, name: "Platform export" }),
).rejects.toBeInstanceOf(InvalidMediaInputError);
expect((await media.listMetricMappings(spaceId)).map((m) => m.name)).toEqual([
"Other",
"Platform export",
]);
expect((await media.getMetricMapping(spaceId, saved.id as string)).delimiter).toBe(",");
expect(await media.getTeaserModel(spaceId)).toBeNull();
const model = await media.putTeaserModel(spaceId, testTeaserWeights, 21);
expect(model).toMatchObject({ sampleCount: 21, weights: testTeaserWeights });
expect((await media.getTeaserModel(spaceId))?.sampleCount).toBe(21);
await media.resetTeaserModel(spaceId);
expect(await media.getTeaserModel(spaceId)).toBeNull();
});
});

View File

@ -0,0 +1,490 @@
import type {
BrandKitRecord,
FfprobeReport,
GateReport,
MediaEditPlan,
MediaProject,
MediaProjectGoal,
MediaProjectStatus,
MediaRender,
MediaRenderOutput,
MediaRenderQuality,
MediaRenderStatus,
} from "@rakazo/contracts";
import { BrandKitSchema, EdlSchema, MAX_AUTO_FIX_ROUNDS } from "@rakazo/contracts";
import type {
MediaProject as MediaProjectRow,
MediaRender as MediaRenderRow,
PrismaClient,
} from "./client.js";
import { Prisma } from "./client.js";
import type { MediaDb, MediaPage, MediaPageInput } from "./media-shared.js";
import {
decodeKeysetCursor,
InvalidMediaInputError,
isUniqueViolation,
MediaStateConflictError,
mapBrandKit,
mapMediaEditPlan,
mapMediaProject,
mapMediaRender,
pageLimit,
toPage,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
export interface NewMediaProject {
title: string;
goal: MediaProjectGoal;
profileIds: string[];
collectionIds?: string[];
assetIds?: string[];
brandKitId?: string | null;
}
export interface MediaProjectPatch {
title?: string;
goal?: MediaProjectGoal;
collectionIds?: string[];
assetIds?: string[];
profileIds?: string[];
brandKitId?: string | null;
status?: MediaProjectStatus;
}
export interface MediaProjectRecord extends MediaProject {
assetIds: string[];
createdBy: string;
}
export interface NewMediaEditPlan {
projectId: string;
profileId: string;
/** Validated against EdlSchema here; semantic validation happens before (validateEdl). */
edl: unknown;
createdBy: "agent" | "user";
rationale: string | null;
}
/** Fields a render worker reports; outputs may carry internal paths next to the public fields. */
export interface MediaRenderPatch {
status?: MediaRenderStatus;
progress?: number;
outputs?: Array<MediaRenderOutput & Record<string, unknown>>;
gateReport?: GateReport | null;
ffprobe?: FfprobeReport | null;
autoFixRound?: number;
error?: string | null;
}
/** Statuses a render can still be changed or canceled from. */
export const ACTIVE_RENDER_STATUSES: MediaRenderStatus[] = ["queued", "rendering", "gating"];
const PLAN_VERSION_ATTEMPTS = 50;
function projectRecord(row: MediaProjectRow): MediaProjectRecord {
return { ...mapMediaProject(row), assetIds: row.assetIds, createdBy: row.createdBy };
}
/** Brand kits, projects, versioned edit plans and renders. */
export function createMediaProjectRepos(prisma: PrismaClient) {
async function assertOwned(
db: MediaDb,
spaceId: string,
model: "mediaBrandKit" | "mediaCollection" | "mediaAsset",
ids: string[],
): Promise<void> {
const unique = [...new Set(ids)];
if (unique.length === 0) return;
const where = { spaceId, id: { in: unique } };
const owned =
model === "mediaBrandKit"
? await db.mediaBrandKit.count({ where })
: model === "mediaCollection"
? await db.mediaCollection.count({ where })
: await db.mediaAsset.count({ where });
if (owned !== unique.length) throw new IsolationError();
}
// -------------------------------------------------------------------------
// Brand kits
// -------------------------------------------------------------------------
async function upsertBrandKit(
spaceId: string,
input: { id?: string; kit: unknown },
): Promise<BrandKitRecord> {
const parsed = BrandKitSchema.safeParse(input.kit);
if (!parsed.success) throw new InvalidMediaInputError("Invalid brand kit");
const kit = parsed.data as Prisma.InputJsonObject;
if (!input.id) {
return mapBrandKit(await prisma.mediaBrandKit.create({ data: { spaceId, kit } }));
}
const result = await prisma.mediaBrandKit.updateMany({
where: { id: input.id, spaceId },
data: { kit },
});
if (result.count !== 1) throw new IsolationError();
return getBrandKit(spaceId, input.id);
}
async function listBrandKits(spaceId: string): Promise<BrandKitRecord[]> {
const rows = await prisma.mediaBrandKit.findMany({
where: { spaceId },
orderBy: [{ updatedAt: "desc" }, { id: "desc" }],
});
return rows.map(mapBrandKit);
}
async function getBrandKit(spaceId: string, id: string): Promise<BrandKitRecord> {
const row = await prisma.mediaBrandKit.findFirst({ where: { id, spaceId } });
if (!row) throw new IsolationError();
return mapBrandKit(row);
}
async function removeBrandKit(spaceId: string, id: string): Promise<void> {
const result = await prisma.mediaBrandKit.deleteMany({ where: { id, spaceId } });
if (result.count !== 1) throw new IsolationError();
}
// -------------------------------------------------------------------------
// Projects and plans
// -------------------------------------------------------------------------
async function requireProject(db: MediaDb, spaceId: string, projectId: string) {
const row = await db.mediaProject.findFirst({ where: { id: projectId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
async function createProject(
spaceId: string,
createdBy: string,
input: NewMediaProject,
): Promise<MediaProject> {
return prisma.$transaction(async (tx) => {
await assertOwned(tx, spaceId, "mediaBrandKit", input.brandKitId ? [input.brandKitId] : []);
await assertOwned(tx, spaceId, "mediaCollection", input.collectionIds ?? []);
await assertOwned(tx, spaceId, "mediaAsset", input.assetIds ?? []);
const row = await tx.mediaProject.create({
data: {
spaceId,
createdBy,
title: input.title,
goal: input.goal,
profileIds: [...new Set(input.profileIds)],
collectionIds: [...new Set(input.collectionIds ?? [])],
assetIds: [...new Set(input.assetIds ?? [])],
brandKitId: input.brandKitId ?? null,
},
});
return mapMediaProject(row);
});
}
async function updateProject(
spaceId: string,
projectId: string,
patch: MediaProjectPatch,
): Promise<MediaProject> {
return prisma.$transaction(async (tx) => {
await assertOwned(tx, spaceId, "mediaBrandKit", patch.brandKitId ? [patch.brandKitId] : []);
await assertOwned(tx, spaceId, "mediaCollection", patch.collectionIds ?? []);
await assertOwned(tx, spaceId, "mediaAsset", patch.assetIds ?? []);
const result = await tx.mediaProject.updateMany({
where: { id: projectId, spaceId },
data: {
...(patch.title !== undefined ? { title: patch.title } : {}),
...(patch.goal !== undefined ? { goal: patch.goal } : {}),
...(patch.status !== undefined ? { status: patch.status } : {}),
...(patch.brandKitId !== undefined ? { brandKitId: patch.brandKitId } : {}),
...(patch.profileIds !== undefined ? { profileIds: [...new Set(patch.profileIds)] } : {}),
...(patch.collectionIds !== undefined
? { collectionIds: [...new Set(patch.collectionIds)] }
: {}),
...(patch.assetIds !== undefined ? { assetIds: [...new Set(patch.assetIds)] } : {}),
},
});
if (result.count !== 1) throw new IsolationError();
return mapMediaProject(await requireProject(tx, spaceId, projectId));
});
}
/** Most recently changed first. */
async function listProjects(
spaceId: string,
input: MediaPageInput = {},
): Promise<MediaPage<MediaProject>> {
const limit = pageLimit(input.limit);
const cursor = input.cursor ? decodeKeysetCursor(input.cursor) : null;
if (cursor && !cursor.at) throw new InvalidMediaInputError("Invalid cursor");
const rows = await prisma.mediaProject.findMany({
where: {
spaceId,
...(cursor?.at
? {
OR: [
{ updatedAt: { lt: cursor.at } },
{ updatedAt: cursor.at, id: { lt: cursor.id } },
],
}
: {}),
},
orderBy: [{ updatedAt: "desc" }, { id: "desc" }],
take: limit + 1,
});
return toPage(rows, limit, (row) => ({ at: row.updatedAt, id: row.id }), mapMediaProject);
}
async function getProjectRecord(spaceId: string, projectId: string): Promise<MediaProjectRecord> {
return projectRecord(await requireProject(prisma, spaceId, projectId));
}
/** Project with the latest plan per profile and its renders, newest first. */
async function getProject(spaceId: string, projectId: string) {
const row = await requireProject(prisma, spaceId, projectId);
const [plans, renders] = await Promise.all([
latestPlans(spaceId, projectId),
listRenders(spaceId, projectId),
]);
return { project: mapMediaProject(row), plans, renders };
}
async function removeProject(spaceId: string, projectId: string): Promise<void> {
const result = await prisma.mediaProject.deleteMany({ where: { id: projectId, spaceId } });
if (result.count !== 1) throw new IsolationError();
}
/**
* Store the next plan version for (project, profile). Concurrent writers race on the
* unique (projectId, profileId, version); the loser re-reads and takes the next number.
*/
async function createPlanVersion(
spaceId: string,
input: NewMediaEditPlan,
): Promise<MediaEditPlan> {
const parsed = EdlSchema.safeParse(input.edl);
if (!parsed.success) throw new InvalidMediaInputError("Invalid edit decision list");
if (parsed.data.output.profileId !== input.profileId) {
throw new InvalidMediaInputError("Plan profile does not match its EDL");
}
await requireProject(prisma, spaceId, input.projectId);
for (let attempt = 1; ; attempt += 1) {
const latest = await prisma.mediaEditPlan.findFirst({
where: { projectId: input.projectId, profileId: input.profileId },
orderBy: { version: "desc" },
select: { version: true },
});
try {
const row = await prisma.mediaEditPlan.create({
data: {
spaceId,
projectId: input.projectId,
profileId: input.profileId,
version: (latest?.version ?? 0) + 1,
edl: parsed.data as Prisma.InputJsonObject,
createdBy: input.createdBy,
rationale: input.rationale,
},
});
return mapMediaEditPlan(row);
} catch (error) {
if (!isUniqueViolation(error) || attempt >= PLAN_VERSION_ATTEMPTS) throw error;
}
}
}
async function getPlan(spaceId: string, planId: string): Promise<MediaEditPlan> {
const row = await prisma.mediaEditPlan.findFirst({ where: { id: planId, spaceId } });
if (!row) throw new IsolationError();
return mapMediaEditPlan(row);
}
async function latestPlan(
spaceId: string,
projectId: string,
profileId: string,
): Promise<MediaEditPlan | null> {
await requireProject(prisma, spaceId, projectId);
const row = await prisma.mediaEditPlan.findFirst({
where: { spaceId, projectId, profileId },
orderBy: { version: "desc" },
});
return row ? mapMediaEditPlan(row) : null;
}
/** The newest plan of every profile in the project, ordered by profile id. */
async function latestPlans(spaceId: string, projectId: string): Promise<MediaEditPlan[]> {
const rows = await prisma.mediaEditPlan.findMany({
where: { spaceId, projectId },
orderBy: [{ profileId: "asc" }, { version: "desc" }],
distinct: ["profileId"],
});
return rows.map(mapMediaEditPlan);
}
async function listPlanVersions(
spaceId: string,
projectId: string,
profileId: string,
): Promise<MediaEditPlan[]> {
await requireProject(prisma, spaceId, projectId);
const rows = await prisma.mediaEditPlan.findMany({
where: { spaceId, projectId, profileId },
orderBy: { version: "desc" },
});
return rows.map(mapMediaEditPlan);
}
// -------------------------------------------------------------------------
// Renders
// -------------------------------------------------------------------------
async function requireRender(db: MediaDb, spaceId: string, renderId: string) {
const row = await db.mediaRender.findFirst({ where: { id: renderId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
async function createRender(
spaceId: string,
input: { planId: string; quality: MediaRenderQuality; createdBy: string },
): Promise<MediaRender> {
const plan = await prisma.mediaEditPlan.findFirst({
where: { id: input.planId, spaceId },
select: { id: true, projectId: true, profileId: true },
});
if (!plan) throw new IsolationError();
const row = await prisma.mediaRender.create({
data: {
spaceId,
projectId: plan.projectId,
planId: plan.id,
profileId: plan.profileId,
quality: input.quality,
createdBy: input.createdBy,
},
});
return mapMediaRender(row);
}
async function getRender(spaceId: string, renderId: string): Promise<MediaRender> {
return mapMediaRender(await requireRender(prisma, spaceId, renderId));
}
/** Full row, including outputs with internal paths. */
async function getRenderRecord(spaceId: string, renderId: string): Promise<MediaRenderRow> {
return requireRender(prisma, spaceId, renderId);
}
async function listRenders(spaceId: string, projectId: string): Promise<MediaRender[]> {
const rows = await prisma.mediaRender.findMany({
where: { spaceId, projectId },
orderBy: [{ createdAt: "desc" }, { id: "desc" }],
});
return rows.map(mapMediaRender);
}
/**
* Take a queued render for processing. The conditional update makes exactly one
* concurrent caller win; the others get null.
*/
async function claimRender(spaceId: string, renderId: string): Promise<MediaRender | null> {
const result = await prisma.mediaRender.updateMany({
where: { id: renderId, spaceId, status: "queued" },
data: { status: "rendering", progress: 0, error: null },
});
const row = await requireRender(prisma, spaceId, renderId);
return result.count === 1 ? mapMediaRender(row) : null;
}
/**
* Apply worker progress or results while the render is in one of `from` (default:
* still active). Returns null when the render already left those states, e.g. after
* a cancel, so a late worker never overwrites it.
*/
async function updateRender(
spaceId: string,
renderId: string,
patch: MediaRenderPatch,
from: MediaRenderStatus[] = ACTIVE_RENDER_STATUSES,
): Promise<MediaRender | null> {
if (
patch.autoFixRound !== undefined &&
(!Number.isInteger(patch.autoFixRound) ||
patch.autoFixRound < 0 ||
patch.autoFixRound > MAX_AUTO_FIX_ROUNDS)
) {
throw new InvalidMediaInputError("Auto-fix round out of range");
}
const result = await prisma.mediaRender.updateMany({
where: { id: renderId, spaceId, status: { in: from } },
data: {
...(patch.status !== undefined ? { status: patch.status } : {}),
...(patch.progress !== undefined
? { progress: Math.min(1, Math.max(0, patch.progress)) }
: {}),
...(patch.outputs !== undefined
? { outputs: patch.outputs as unknown as Prisma.InputJsonArray }
: {}),
...(patch.gateReport !== undefined
? {
gateReport:
patch.gateReport === null
? Prisma.DbNull
: (patch.gateReport as unknown as Prisma.InputJsonObject),
}
: {}),
...(patch.ffprobe !== undefined
? {
ffprobe:
patch.ffprobe === null
? Prisma.DbNull
: (patch.ffprobe as unknown as Prisma.InputJsonObject),
}
: {}),
...(patch.autoFixRound !== undefined ? { autoFixRound: patch.autoFixRound } : {}),
...(patch.error !== undefined ? { error: patch.error } : {}),
},
});
const row = await requireRender(prisma, spaceId, renderId);
return result.count === 1 ? mapMediaRender(row) : null;
}
async function cancelRender(spaceId: string, renderId: string): Promise<MediaRender> {
const result = await prisma.mediaRender.updateMany({
where: { id: renderId, spaceId, status: { in: ACTIVE_RENDER_STATUSES } },
data: { status: "canceled" },
});
const row = await requireRender(prisma, spaceId, renderId);
if (result.count !== 1) {
throw new MediaStateConflictError(`Render is already ${row.status}`);
}
return mapMediaRender(row);
}
return {
upsertBrandKit,
listBrandKits,
getBrandKit,
removeBrandKit,
createProject,
updateProject,
listProjects,
getProject,
getProjectRecord,
removeProject,
createPlanVersion,
getPlan,
latestPlan,
latestPlans,
listPlanVersions,
createRender,
getRender,
getRenderRecord,
listRenders,
claimRender,
updateRender,
cancelRender,
};
}

View File

@ -0,0 +1,497 @@
import type {
FunnelRow,
MediaMetric,
MediaPublication,
MediaPublicationStatus,
MetricCsvMapping,
TeaserWeights,
Utm,
} from "@rakazo/contracts";
import { MetricCsvMappingSchema, TeaserWeightsSchema } from "@rakazo/contracts";
import type { MediaPublication as MediaPublicationRow, PrismaClient } from "./client.js";
import { Prisma } from "./client.js";
import type { MediaDb, MediaPage, MediaPageInput } from "./media-shared.js";
import {
decodeKeysetCursor,
InvalidMediaInputError,
isUniqueViolation,
MediaStateConflictError,
mapMediaMetric,
mapMediaPublication,
mapMetricMapping,
pageLimit,
toPage,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
export interface NewMediaPublication {
renderId: string;
destination: string;
targetUrl: string | null;
trackingUrl: string | null;
utm: Utm | null;
title: string | null;
caption: string | null;
tags: string[];
thumbnailVariant: string | null;
scheduledAt: Date | null;
}
export type PublicationKey = MetricCsvMapping["publicationKey"];
export type NewMediaMetric = Omit<MediaMetric, "id">;
export interface TeaserModelRecord {
weights: TeaserWeights;
sampleCount: number;
updatedAt: string;
}
function assertMetric(metric: NewMediaMetric): void {
for (const value of [metric.views, metric.clicks, metric.conversions, metric.revenueCents]) {
if (!Number.isInteger(value) || value < 0) {
throw new InvalidMediaInputError("Metric values must be non-negative integers");
}
}
if (!/^[A-Z]{3}$/.test(metric.currency)) throw new InvalidMediaInputError("Invalid currency");
}
/** Publications, their metrics and the learned teaser weights. */
export function createMediaPublishingRepos(prisma: PrismaClient) {
async function requireRender(spaceId: string, renderId: string) {
const row = await prisma.mediaRender.findFirst({ where: { id: renderId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
async function requireProject(spaceId: string, projectId: string): Promise<void> {
const row = await prisma.mediaProject.findFirst({
where: { id: projectId, spaceId },
select: { id: true },
});
if (!row) throw new IsolationError();
}
async function requirePublication(spaceId: string, publicationId: string) {
const row = await prisma.mediaPublication.findFirst({ where: { id: publicationId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
/** Only a finished render that passed its gate can be published. */
async function createPublication(
spaceId: string,
createdBy: string,
input: NewMediaPublication,
): Promise<MediaPublication> {
const render = await requireRender(spaceId, input.renderId);
if (render.status !== "done") {
throw new MediaStateConflictError("Only a finished render can be published");
}
const row = await prisma.mediaPublication.create({
data: {
spaceId,
createdBy,
renderId: render.id,
profileId: render.profileId,
destination: input.destination,
status: input.scheduledAt ? "scheduled" : "draft",
scheduledAt: input.scheduledAt,
targetUrl: input.targetUrl,
trackingUrl: input.trackingUrl,
utm: input.utm === null ? Prisma.DbNull : (input.utm as Prisma.InputJsonObject),
title: input.title,
caption: input.caption,
tags: input.tags,
thumbnailVariant: input.thumbnailVariant,
},
});
return mapMediaPublication(row);
}
async function getPublication(spaceId: string, publicationId: string): Promise<MediaPublication> {
return mapMediaPublication(await requirePublication(spaceId, publicationId));
}
async function getPublicationRecord(
spaceId: string,
publicationId: string,
): Promise<MediaPublicationRow> {
return requirePublication(spaceId, publicationId);
}
async function listPublications(
spaceId: string,
input: MediaPageInput & { projectId?: string } = {},
): Promise<MediaPage<MediaPublication>> {
const limit = pageLimit(input.limit);
const cursor = input.cursor ? decodeKeysetCursor(input.cursor) : null;
if (cursor && !cursor.at) throw new InvalidMediaInputError("Invalid cursor");
const rows = await prisma.mediaPublication.findMany({
where: {
spaceId,
...(input.projectId ? { render: { projectId: input.projectId, spaceId } } : {}),
...(cursor?.at
? {
OR: [
{ createdAt: { lt: cursor.at } },
{ createdAt: cursor.at, id: { lt: cursor.id } },
],
}
: {}),
},
orderBy: [{ createdAt: "desc" }, { id: "desc" }],
take: limit + 1,
});
return toPage(rows, limit, (row) => ({ at: row.createdAt, id: row.id }), mapMediaPublication);
}
async function transitionPublication(
spaceId: string,
publicationId: string,
from: MediaPublicationStatus[],
data: Prisma.MediaPublicationUpdateManyMutationInput,
): Promise<MediaPublication> {
const result = await prisma.mediaPublication.updateMany({
where: { id: publicationId, spaceId, status: { in: from } },
data,
});
const row = await requirePublication(spaceId, publicationId);
if (result.count !== 1) {
throw new MediaStateConflictError(`Publication is already ${row.status}`);
}
return mapMediaPublication(row);
}
async function schedulePublication(
spaceId: string,
publicationId: string,
scheduledAt: Date,
): Promise<MediaPublication> {
return transitionPublication(spaceId, publicationId, ["draft", "scheduled", "failed"], {
status: "scheduled",
scheduledAt,
});
}
async function cancelPublication(
spaceId: string,
publicationId: string,
): Promise<MediaPublication> {
return transitionPublication(
spaceId,
publicationId,
["draft", "scheduled", "bundled", "failed"],
{ status: "canceled" },
);
}
async function markPublicationBundled(
spaceId: string,
publicationId: string,
bundleArtifactId: string | null,
): Promise<MediaPublication> {
return transitionPublication(
spaceId,
publicationId,
["draft", "scheduled", "bundled", "failed"],
{ status: "bundled", bundleArtifactId },
);
}
async function markPublicationPublished(
spaceId: string,
publicationId: string,
externalUrl: string | null,
publishedAt: Date = new Date(),
): Promise<MediaPublication> {
return transitionPublication(spaceId, publicationId, ["draft", "scheduled", "bundled"], {
status: "published",
externalUrl,
publishedAt,
});
}
async function markPublicationFailed(
spaceId: string,
publicationId: string,
): Promise<MediaPublication> {
return transitionPublication(spaceId, publicationId, ["draft", "scheduled", "bundled"], {
status: "failed",
});
}
/**
* Map CSV keys to publication ids of this space. Values that match nothing (or a
* publication in another space) are absent from the result.
*/
async function resolvePublicationKeys(
spaceId: string,
key: PublicationKey,
values: string[],
): Promise<Map<string, string>> {
const wanted = new Set(values.map((value) => value.trim()).filter(Boolean));
if (wanted.size === 0) return new Map();
const rows = await prisma.mediaPublication.findMany({
where: {
spaceId,
...(key === "id" ? { id: { in: [...wanted] } } : {}),
...(key === "tracking_url" ? { trackingUrl: { in: [...wanted] } } : {}),
...(key === "external_url" ? { externalUrl: { in: [...wanted] } } : {}),
},
select: { id: true, trackingUrl: true, externalUrl: true, utm: true },
});
const resolved = new Map<string, string>();
for (const row of rows) {
const utm = row.utm as { content?: unknown } | null;
const value =
key === "id"
? row.id
: key === "tracking_url"
? row.trackingUrl
: key === "external_url"
? row.externalUrl
: typeof utm?.content === "string"
? utm.content
: null;
if (value && wanted.has(value) && !resolved.has(value)) resolved.set(value, row.id);
}
return resolved;
}
// -------------------------------------------------------------------------
// Metrics
// -------------------------------------------------------------------------
async function upsertMetric(db: MediaDb, spaceId: string, metric: NewMediaMetric) {
assertMetric(metric);
const observedAt = new Date(metric.observedAt);
if (Number.isNaN(observedAt.getTime())) throw new InvalidMediaInputError("Invalid date");
const values = {
views: metric.views,
clicks: metric.clicks,
conversions: metric.conversions,
revenueCents: metric.revenueCents,
currency: metric.currency,
};
return db.mediaMetric.upsert({
where: {
publicationId_observedAt_source: {
publicationId: metric.publicationId,
observedAt,
source: metric.source,
},
},
create: {
...values,
spaceId,
publicationId: metric.publicationId,
observedAt,
source: metric.source,
},
update: values,
});
}
/** One observation; re-recording the same (publication, time, source) replaces it. */
async function recordMetric(spaceId: string, metric: NewMediaMetric): Promise<MediaMetric> {
await requirePublication(spaceId, metric.publicationId);
return mapMediaMetric(await upsertMetric(prisma, spaceId, metric));
}
/** Batch import, all or nothing. */
async function recordMetrics(spaceId: string, metrics: NewMediaMetric[]): Promise<number> {
if (metrics.length === 0) return 0;
return prisma.$transaction(
async (tx) => {
const ids = [...new Set(metrics.map((metric) => metric.publicationId))];
const owned = await tx.mediaPublication.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
for (const metric of metrics) await upsertMetric(tx, spaceId, metric);
return metrics.length;
},
{ timeout: 60_000 },
);
}
async function listMetrics(spaceId: string, publicationId: string): Promise<MediaMetric[]> {
await requirePublication(spaceId, publicationId);
const rows = await prisma.mediaMetric.findMany({
where: { spaceId, publicationId },
orderBy: [{ observedAt: "asc" }, { source: "asc" }],
});
return rows.map(mapMediaMetric);
}
async function saveMetricMapping(
spaceId: string,
input: MetricCsvMapping,
): Promise<MetricCsvMapping> {
const parsed = MetricCsvMappingSchema.safeParse(input);
if (!parsed.success) throw new InvalidMediaInputError("Invalid CSV mapping");
const { id, ...mapping } = parsed.data;
const json = mapping as Prisma.InputJsonObject;
try {
if (id) {
const result = await prisma.mediaMetricMapping.updateMany({
where: { id, spaceId },
data: { name: mapping.name, mapping: json },
});
if (result.count !== 1) throw new IsolationError();
return getMetricMapping(spaceId, id);
}
const row = await prisma.mediaMetricMapping.upsert({
where: { spaceId_name: { spaceId, name: mapping.name } },
create: { spaceId, name: mapping.name, mapping: json },
update: { mapping: json },
});
return mapMetricMapping(row);
} catch (error) {
if (isUniqueViolation(error)) throw new InvalidMediaInputError("Mapping name already used");
throw error;
}
}
async function listMetricMappings(spaceId: string): Promise<MetricCsvMapping[]> {
const rows = await prisma.mediaMetricMapping.findMany({
where: { spaceId },
orderBy: [{ name: "asc" }, { id: "asc" }],
});
return rows.map(mapMetricMapping);
}
async function getMetricMapping(spaceId: string, id: string): Promise<MetricCsvMapping> {
const row = await prisma.mediaMetricMapping.findFirst({ where: { id, spaceId } });
if (!row) throw new IsolationError();
return mapMetricMapping(row);
}
/**
* Views → clicks → conversions → revenue per publication. Observations are
* cumulative snapshots, so only the latest one per (publication, source) counts;
* importing the same report twice never doubles the numbers.
*/
async function funnel(spaceId: string, projectId?: string): Promise<FunnelRow[]> {
if (projectId) await requireProject(spaceId, projectId);
const rows = await prisma.$queryRaw<
Array<{
publicationId: string;
projectId: string;
profileId: string;
variant: string | null;
thumbnailVariant: string | null;
views: bigint;
clicks: bigint;
conversions: bigint;
revenueCents: bigint;
currency: string | null;
}>
>(Prisma.sql`
WITH latest AS (
SELECT DISTINCT ON (m."publicationId", m.source)
m."publicationId", m.views, m.clicks, m.conversions, m."revenueCents", m.currency
FROM media_metrics m
WHERE m."spaceId" = ${spaceId}
ORDER BY m."publicationId", m.source, m."observedAt" DESC, m.id DESC
)
SELECT p.id AS "publicationId", r."projectId", p."profileId",
p.utm ->> 'content' AS variant, p."thumbnailVariant",
coalesce(sum(l.views), 0)::bigint AS views,
coalesce(sum(l.clicks), 0)::bigint AS clicks,
coalesce(sum(l.conversions), 0)::bigint AS conversions,
coalesce(sum(l."revenueCents"), 0)::bigint AS "revenueCents",
min(l.currency) AS currency
FROM media_publications p
JOIN media_renders r ON r.id = p."renderId" AND r."spaceId" = p."spaceId"
LEFT JOIN latest l ON l."publicationId" = p.id
WHERE p."spaceId" = ${spaceId}
${projectId ? Prisma.sql`AND r."projectId" = ${projectId}` : Prisma.empty}
GROUP BY p.id, r."projectId"
ORDER BY p."createdAt" DESC, p.id DESC
`);
return rows.map((row) => ({
publicationId: row.publicationId,
projectId: row.projectId,
profileId: row.profileId,
variant: row.variant,
thumbnailVariant: row.thumbnailVariant,
views: Number(row.views),
clicks: Number(row.clicks),
conversions: Number(row.conversions),
revenueCents: Number(row.revenueCents),
currency: row.currency ?? "EUR",
}));
}
/** Publications with at least one metric: the sample size for teaser learning. */
async function publicationsWithMetricsCount(spaceId: string): Promise<number> {
return prisma.mediaPublication.count({ where: { spaceId, metrics: { some: {} } } });
}
// -------------------------------------------------------------------------
// Teaser model
// -------------------------------------------------------------------------
async function getTeaserModel(spaceId: string): Promise<TeaserModelRecord | null> {
const row = await prisma.mediaTeaserModel.findUnique({ where: { spaceId } });
if (!row) return null;
const weights = TeaserWeightsSchema.safeParse(row.weights);
if (!weights.success) return null;
return {
weights: weights.data,
sampleCount: row.sampleCount,
updatedAt: row.updatedAt.toISOString(),
};
}
async function putTeaserModel(
spaceId: string,
weights: TeaserWeights,
sampleCount: number,
): Promise<TeaserModelRecord> {
const parsed = TeaserWeightsSchema.safeParse(weights);
if (!parsed.success || !Number.isInteger(sampleCount) || sampleCount < 0) {
throw new InvalidMediaInputError("Invalid teaser model");
}
const json = parsed.data as Prisma.InputJsonObject;
const row = await prisma.mediaTeaserModel.upsert({
where: { spaceId },
create: { spaceId, weights: json, sampleCount },
update: { weights: json, sampleCount },
});
return {
weights: parsed.data,
sampleCount: row.sampleCount,
updatedAt: row.updatedAt.toISOString(),
};
}
/** Back to default weights. */
async function resetTeaserModel(spaceId: string): Promise<void> {
await prisma.mediaTeaserModel.deleteMany({ where: { spaceId } });
}
return {
createPublication,
getPublication,
getPublicationRecord,
listPublications,
schedulePublication,
cancelPublication,
markPublicationBundled,
markPublicationPublished,
markPublicationFailed,
resolvePublicationKeys,
recordMetric,
recordMetrics,
listMetrics,
saveMetricMapping,
listMetricMappings,
getMetricMapping,
funnel,
publicationsWithMetricsCount,
getTeaserModel,
putTeaserModel,
resetTeaserModel,
};
}

View File

@ -0,0 +1,526 @@
import { createHash } from "node:crypto";
import type { MediaSourceItem } from "@rakazo/adapter-kit";
import type {
AgeVerificationMethod,
AnonymizeMode,
BrandKitRecord,
ClientMediaItem,
MarkTreatment,
MediaAsset,
MediaAssetKind,
MediaAssetStorage,
MediaCollection,
MediaCollectionKind,
MediaEditPlan,
MediaFaceTrack,
MediaIngestState,
MediaLabelSource,
MediaMark,
MediaMarkKind,
MediaMetric,
MediaMetricSource,
MediaPerson,
MediaProject,
MediaProjectGoal,
MediaProjectStatus,
MediaPublication,
MediaPublicationStatus,
MediaRender,
MediaRenderOutput,
MediaRenderQuality,
MediaRenderStatus,
MediaSegment,
MediaSegmentKind,
MediaSource,
MediaSourceKind,
MediaSourceStatus,
MediaUpload,
MetricCsvMapping,
TeaserFeature,
} from "@rakazo/contracts";
import {
BrandKitSchema,
EdlSchema,
FfprobeReportSchema,
GateReportSchema,
MediaMarkPromptSchema,
MediaRenderOutputSchema,
MetricCsvMappingSchema,
UtmSchema,
} from "@rakazo/contracts";
import type {
MediaAsset as MediaAssetRow,
MediaBrandKit as MediaBrandKitRow,
MediaCollection as MediaCollectionRow,
MediaEditPlan as MediaEditPlanRow,
MediaFaceTrack as MediaFaceTrackRow,
MediaMark as MediaMarkRow,
MediaMetricMapping as MediaMetricMappingRow,
MediaMetric as MediaMetricRow,
MediaPerson as MediaPersonRow,
MediaProject as MediaProjectRow,
MediaPublication as MediaPublicationRow,
MediaRender as MediaRenderRow,
MediaSegment as MediaSegmentRow,
MediaSource as MediaSourceRow,
MediaUpload as MediaUploadRow,
PrismaClient,
} from "./client.js";
import { Prisma } from "./client.js";
import { IsolationError } from "./scope.js";
/*
* Shared pieces of the Media Studio repositories: errors, keyset cursors, and
* the row → DTO mappers. Not-found and foreign-space access both surface as
* IsolationError (from scope.ts), like every other repository.
*/
/** Input the repository refuses: bad cursor, invalid EDL or brand kit, foreign ids. */
export class InvalidMediaInputError extends Error {
constructor(message: string) {
super(message);
this.name = "InvalidMediaInputError";
}
}
/** The record exists but its state does not allow the requested change. */
export class MediaStateConflictError extends Error {
constructor(message: string) {
super(message);
this.name = "MediaStateConflictError";
}
}
export type MediaDb = Prisma.TransactionClient | PrismaClient;
/** Throws IsolationError unless every id is an asset of the space. */
export async function assertAssetsInSpace(
db: MediaDb,
spaceId: string,
assetIds: string[],
): Promise<void> {
const ids = [...new Set(assetIds)];
if (ids.length === 0) return;
const owned = await db.mediaAsset.count({ where: { spaceId, id: { in: ids } } });
if (owned !== ids.length) throw new IsolationError();
}
export const MEDIA_PAGE_MAX = 500;
export const MEDIA_PAGE_DEFAULT = 100;
export interface MediaPage<T> {
items: T[];
nextCursor: string | null;
}
export interface MediaPageInput {
cursor?: string | null;
limit?: number;
}
export function pageLimit(limit: number | undefined): number {
if (limit === undefined || !Number.isFinite(limit)) return MEDIA_PAGE_DEFAULT;
return Math.min(MEDIA_PAGE_MAX, Math.max(1, Math.trunc(limit)));
}
/**
* Position after the last row of a page ordered by `(at DESC NULLS LAST, id DESC)`.
* `at` is null for rows without a timestamp, which sort after every dated row.
*/
export interface KeysetCursor {
at: Date | null;
id: string;
}
/** Opaque base64url token for a keyset position. */
export function encodeKeysetCursor(cursor: KeysetCursor): string {
return Buffer.from(JSON.stringify([cursor.at?.toISOString() ?? null, cursor.id])).toString(
"base64url",
);
}
export function decodeKeysetCursor(token: string): KeysetCursor {
let parsed: unknown;
try {
parsed = JSON.parse(Buffer.from(token, "base64url").toString("utf8"));
} catch {
throw new InvalidMediaInputError("Invalid cursor");
}
if (!Array.isArray(parsed) || parsed.length !== 2) {
throw new InvalidMediaInputError("Invalid cursor");
}
const [at, id] = parsed as unknown[];
if (typeof id !== "string" || id.length === 0 || id.length > 100) {
throw new InvalidMediaInputError("Invalid cursor");
}
if (at === null) return { at: null, id };
if (typeof at !== "string") throw new InvalidMediaInputError("Invalid cursor");
const date = new Date(at);
if (Number.isNaN(date.getTime())) throw new InvalidMediaInputError("Invalid cursor");
return { at: date, id };
}
/**
* SQL predicate for "after the cursor" under `ORDER BY at DESC NULLS LAST, id DESC`.
* `atColumn` and `idColumn` are trusted column references.
*/
export function keysetAfterSql(
cursor: KeysetCursor | null,
atColumn: Prisma.Sql,
idColumn: Prisma.Sql,
): Prisma.Sql {
if (!cursor) return Prisma.sql`TRUE`;
if (cursor.at === null) {
return Prisma.sql`(${atColumn} IS NULL AND ${idColumn} < ${cursor.id})`;
}
return Prisma.sql`(${atColumn} < ${cursor.at} OR (${atColumn} = ${cursor.at} AND ${idColumn} < ${cursor.id}) OR ${atColumn} IS NULL)`;
}
/** Slice a `limit + 1` result into a page with its next cursor. */
export function toPage<Row, Item>(
rows: Row[],
limit: number,
keyOf: (row: Row) => KeysetCursor,
map: (row: Row) => Item,
): MediaPage<Item> {
const hasMore = rows.length > limit;
const pageRows = hasMore ? rows.slice(0, limit) : rows;
const last = pageRows.at(-1);
return {
items: pageRows.map(map),
nextCursor: hasMore && last ? encodeKeysetCursor(keyOf(last)) : null,
};
}
/** Escape LIKE wildcards so user text matches literally. */
export function likeContains(text: string): string {
return `%${text.replace(/[\\%_]/g, (char) => `\\${char}`)}%`;
}
export function isUniqueViolation(error: unknown): boolean {
return Boolean(error && typeof error === "object" && "code" in error && error.code === "P2002");
}
/** Advisory lock held until the surrounding transaction ends. */
export async function lockMediaKey(tx: Prisma.TransactionClient, key: string): Promise<void> {
await tx.$queryRaw(Prisma.sql`
SELECT pg_advisory_xact_lock(hashtextextended(${`media:${key}`}, 0))::text AS "lock"
`);
}
/** Prisma Bytes columns want an ArrayBuffer-backed view; copy when the input is a slice. */
export function toBytes(value: Uint8Array): Uint8Array<ArrayBuffer> {
return new Uint8Array(value);
}
/**
* Change detector for a client-pushed item, which has no revision of its own:
* any metadata change produces a different value.
*/
export function clientMediaItemToSourceItem(item: ClientMediaItem): MediaSourceItem {
const canonical = JSON.stringify([
item.kind,
item.capturedAt,
item.durationMs,
item.width,
item.height,
item.fps,
item.gpsLat,
item.gpsLon,
item.deviceModel,
item.albumNames,
item.isFavorite,
item.fileName,
]);
return {
externalId: item.externalId,
kind: item.kind,
capturedAt: item.capturedAt ? new Date(item.capturedAt) : null,
durationMs: item.durationMs,
width: item.width,
height: item.height,
fps: item.fps,
gpsLat: item.gpsLat,
gpsLon: item.gpsLon,
deviceModel: item.deviceModel,
albumNames: item.albumNames,
isFavorite: item.isFavorite,
fileName: item.fileName,
sizeBytes: null,
revision: createHash("sha256").update(canonical).digest("hex").slice(0, 32),
};
}
// ---------------------------------------------------------------------------
// Mappers
// ---------------------------------------------------------------------------
function iso(value: Date | null): string | null {
return value ? value.toISOString() : null;
}
export function mapMediaSource(row: MediaSourceRow, assetCount: number): MediaSource {
return {
id: row.id,
kind: row.kind as MediaSourceKind,
label: row.label,
status: row.status as MediaSourceStatus,
assetCount,
lastSyncedAt: iso(row.lastSyncedAt),
error: row.error,
createdAt: row.createdAt.toISOString(),
};
}
export function mapMediaAsset(row: MediaAssetRow): MediaAsset {
return {
id: row.id,
sourceId: row.sourceId,
externalId: row.externalId,
kind: row.kind as MediaAssetKind,
storage: row.storage as MediaAssetStorage,
capturedAt: iso(row.capturedAt),
durationMs: row.durationMs,
width: row.width,
height: row.height,
fps: row.fps,
gpsLat: row.gpsLat,
gpsLon: row.gpsLon,
deviceModel: row.deviceModel,
albumNames: row.albumNames,
isFavorite: row.isFavorite,
ingestState: row.ingestState as MediaIngestState,
hasProxy: row.proxyPath !== null,
hasThumbnail: row.thumbPath !== null,
qualityScore: row.qualityScore,
createdAt: row.createdAt.toISOString(),
};
}
function numberRecord(value: Prisma.JsonValue): Partial<Record<TeaserFeature, number>> {
if (!value || typeof value !== "object" || Array.isArray(value)) return {};
const out: Partial<Record<TeaserFeature, number>> = {};
for (const [key, score] of Object.entries(value)) {
if (typeof score === "number" && Number.isFinite(score)) {
out[key as TeaserFeature] = score;
}
}
return out;
}
export function mapMediaSegment(row: MediaSegmentRow): MediaSegment {
return {
id: row.id,
assetId: row.assetId,
kind: row.kind as MediaSegmentKind,
startMs: row.startMs,
endMs: row.endMs,
scores: numberRecord(row.scores),
teaserScore: row.teaserScore,
transcriptExcerpt: row.transcriptExcerpt,
labels: row.labels,
paidOnly: row.paidOnly,
};
}
export interface CollectionStats {
assetCount: number;
videoCount: number;
totalDurationMs: number;
}
export const EMPTY_COLLECTION_STATS: CollectionStats = {
assetCount: 0,
videoCount: 0,
totalDurationMs: 0,
};
export function mapMediaCollection(
row: MediaCollectionRow,
stats: CollectionStats,
): MediaCollection {
return {
id: row.id,
kind: row.kind as MediaCollectionKind,
title: row.title,
labelSource: row.labelSource as MediaLabelSource,
startAt: iso(row.startAt),
endAt: iso(row.endAt),
coverAssetId: row.coverAssetId,
assetCount: stats.assetCount,
videoCount: stats.videoCount,
totalDurationMs: stats.totalDurationMs,
};
}
export function mapMediaPerson(
row: MediaPersonRow,
faceTrackCount: number,
coverThumbAssetId: string | null,
): MediaPerson {
return {
id: row.id,
displayName: row.displayName,
isSelf: row.isSelf,
confirmed: row.confirmed,
anonymizeByDefault: row.anonymizeByDefault as AnonymizeMode,
consentRecordedAt: iso(row.consentRecordedAt),
consentArtifactId: row.consentArtifactId,
ageVerifiedAt: iso(row.ageVerifiedAt),
ageVerificationMethod: row.ageVerificationMethod as AgeVerificationMethod,
faceTrackCount,
coverThumbAssetId,
};
}
export function mapMediaFaceTrack(
row: Pick<
MediaFaceTrackRow,
"id" | "assetId" | "personId" | "startMs" | "endMs" | "confidenceMin"
>,
): MediaFaceTrack {
return {
id: row.id,
assetId: row.assetId,
personId: row.personId,
startMs: row.startMs,
endMs: row.endMs,
confidenceMin: row.confidenceMin,
};
}
export function mapMediaMark(row: MediaMarkRow): MediaMark {
const prompt = row.prompt === null ? null : MediaMarkPromptSchema.safeParse(row.prompt);
return {
id: row.id,
kind: row.kind as MediaMarkKind,
label: row.label,
personId: row.personId,
assetId: row.assetId,
prompt: prompt?.success ? prompt.data : null,
treatment: row.treatment as MarkTreatment,
qualityScore: row.qualityScore,
suggestedFallback: row.suggestedFallback as MarkTreatment | null,
createdAt: row.createdAt.toISOString(),
};
}
export function mapBrandKit(row: MediaBrandKitRow): BrandKitRecord {
return {
id: row.id,
kit: BrandKitSchema.parse(row.kit),
createdAt: row.createdAt.toISOString(),
updatedAt: row.updatedAt.toISOString(),
};
}
export function mapMediaProject(row: MediaProjectRow): MediaProject {
return {
id: row.id,
title: row.title,
goal: row.goal as MediaProjectGoal,
status: row.status as MediaProjectStatus,
brandKitId: row.brandKitId,
collectionIds: row.collectionIds,
profileIds: row.profileIds,
createdAt: row.createdAt.toISOString(),
updatedAt: row.updatedAt.toISOString(),
};
}
export function mapMediaEditPlan(row: MediaEditPlanRow): MediaEditPlan {
return {
id: row.id,
projectId: row.projectId,
profileId: row.profileId,
version: row.version,
edl: EdlSchema.parse(row.edl),
createdBy: row.createdBy === "agent" ? "agent" : "user",
rationale: row.rationale,
createdAt: row.createdAt.toISOString(),
};
}
/** Public render outputs; internal paths stored next to them are dropped. */
export function renderOutputsOf(value: Prisma.JsonValue): MediaRenderOutput[] {
if (!Array.isArray(value)) return [];
const outputs: MediaRenderOutput[] = [];
for (const entry of value) {
const parsed = MediaRenderOutputSchema.safeParse(entry);
if (parsed.success) outputs.push(parsed.data);
}
return outputs;
}
export function mapMediaRender(row: MediaRenderRow): MediaRender {
const gate = row.gateReport === null ? null : GateReportSchema.safeParse(row.gateReport);
const probe = row.ffprobe === null ? null : FfprobeReportSchema.safeParse(row.ffprobe);
return {
id: row.id,
planId: row.planId,
projectId: row.projectId,
profileId: row.profileId,
quality: row.quality as MediaRenderQuality,
status: row.status as MediaRenderStatus,
progress: Math.min(1, Math.max(0, row.progress)),
outputs: renderOutputsOf(row.outputs),
gateReport: gate?.success ? gate.data : null,
ffprobe: probe?.success ? probe.data : null,
error: row.error,
createdAt: row.createdAt.toISOString(),
updatedAt: row.updatedAt.toISOString(),
};
}
export function mapMediaUpload(row: MediaUploadRow): MediaUpload {
return {
id: row.id,
assetId: row.assetId,
sizeBytes: Number(row.sizeBytes),
chunkBytes: row.chunkBytes,
receivedChunks: [...row.receivedChunks].sort((a, b) => a - b),
status: row.status as MediaUpload["status"],
};
}
export function mapMediaPublication(row: MediaPublicationRow): MediaPublication {
const utm = row.utm === null ? null : UtmSchema.safeParse(row.utm);
return {
id: row.id,
renderId: row.renderId,
profileId: row.profileId,
destination: row.destination,
status: row.status as MediaPublicationStatus,
scheduledAt: iso(row.scheduledAt),
publishedAt: iso(row.publishedAt),
externalUrl: row.externalUrl,
targetUrl: row.targetUrl,
trackingUrl: row.trackingUrl,
utm: utm?.success ? utm.data : null,
title: row.title,
caption: row.caption,
tags: row.tags,
thumbnailVariant: row.thumbnailVariant,
bundleArtifactId: row.bundleArtifactId,
createdAt: row.createdAt.toISOString(),
};
}
export function mapMediaMetric(row: MediaMetricRow): MediaMetric {
return {
id: row.id,
publicationId: row.publicationId,
observedAt: row.observedAt.toISOString(),
views: row.views,
clicks: row.clicks,
conversions: row.conversions,
revenueCents: row.revenueCents,
currency: row.currency,
source: row.source as MediaMetricSource,
};
}
export function mapMetricMapping(row: MediaMetricMappingRow): MetricCsvMapping {
const mapping = MetricCsvMappingSchema.parse(row.mapping);
return { ...mapping, id: row.id, name: row.name };
}

View File

@ -0,0 +1,163 @@
import type { MediaUpload } from "@rakazo/contracts";
import { MEDIA_UPLOAD_CHUNK_BYTES } from "@rakazo/contracts";
import type { PrismaClient } from "./client.js";
import { Prisma } from "./client.js";
import {
InvalidMediaInputError,
lockMediaKey,
MediaStateConflictError,
mapMediaUpload,
} from "./media-shared.js";
import { IsolationError } from "./scope.js";
export interface BeginMediaUpload {
assetId: string;
userId: string;
sizeBytes: number;
sha256: string;
fileName: string;
chunkBytes?: number;
}
function chunkCountOf(row: { sizeBytes: bigint; chunkBytes: number }): number {
return Number((row.sizeBytes + BigInt(row.chunkBytes) - 1n) / BigInt(row.chunkBytes));
}
/** Resumable chunked uploads of originals from client-pushed sources. */
export function createMediaUploadRepos(prisma: PrismaClient) {
/** Opens an upload, or resumes the open one for the same asset and content. */
async function beginUpload(spaceId: string, input: BeginMediaUpload): Promise<MediaUpload> {
const chunkBytes = input.chunkBytes ?? MEDIA_UPLOAD_CHUNK_BYTES;
if (!Number.isInteger(input.sizeBytes) || input.sizeBytes <= 0) {
throw new InvalidMediaInputError("Upload size must be positive");
}
if (!Number.isInteger(chunkBytes) || chunkBytes <= 0) {
throw new InvalidMediaInputError("Chunk size must be positive");
}
return prisma.$transaction(async (tx) => {
const asset = await tx.mediaAsset.findFirst({
where: { id: input.assetId, spaceId },
select: { id: true },
});
if (!asset) throw new IsolationError();
await lockMediaKey(tx, `upload:${input.assetId}`);
const open = await tx.mediaUpload.findFirst({
where: {
spaceId,
assetId: input.assetId,
status: "open",
sha256: input.sha256,
sizeBytes: BigInt(input.sizeBytes),
chunkBytes,
},
orderBy: [{ createdAt: "desc" }, { id: "desc" }],
});
if (open) return mapMediaUpload(open);
const row = await tx.mediaUpload.create({
data: {
spaceId,
assetId: input.assetId,
userId: input.userId,
sizeBytes: BigInt(input.sizeBytes),
chunkBytes,
sha256: input.sha256,
fileName: input.fileName,
},
});
return mapMediaUpload(row);
});
}
async function requireUpload(spaceId: string, uploadId: string) {
const row = await prisma.mediaUpload.findFirst({ where: { id: uploadId, spaceId } });
if (!row) throw new IsolationError();
return row;
}
async function uploadStatus(spaceId: string, uploadId: string): Promise<MediaUpload> {
return mapMediaUpload(await requireUpload(spaceId, uploadId));
}
/** Full row (sha256, file name, owner) for the upload route. */
async function getUploadRecord(spaceId: string, uploadId: string) {
return requireUpload(spaceId, uploadId);
}
/** Idempotent: the index is appended once, atomically, however often it is reported. */
async function markChunkReceived(
spaceId: string,
uploadId: string,
index: number,
): Promise<MediaUpload> {
const row = await requireUpload(spaceId, uploadId);
if (!Number.isInteger(index) || index < 0 || index >= chunkCountOf(row)) {
throw new InvalidMediaInputError("Chunk index out of range");
}
if (row.status !== "open") throw new MediaStateConflictError("Upload is not open");
await prisma.$executeRaw(Prisma.sql`
UPDATE media_uploads
SET "receivedChunks" = array_append("receivedChunks", ${index}::int), "updatedAt" = now()
WHERE id = ${uploadId} AND "spaceId" = ${spaceId} AND status = 'open'
AND NOT (${index}::int = ANY("receivedChunks"))
`);
return uploadStatus(spaceId, uploadId);
}
/**
* Closes a fully received upload and points the asset at the stored original. The
* caller has verified the assembled file against `sha256` before calling.
*/
async function completeUpload(
spaceId: string,
uploadId: string,
stored: { storagePath: string },
): Promise<MediaUpload> {
return prisma.$transaction(async (tx) => {
const row = await tx.mediaUpload.findFirst({ where: { id: uploadId, spaceId } });
if (!row) throw new IsolationError();
if (row.status === "complete") return mapMediaUpload(row);
if (row.status !== "open") throw new MediaStateConflictError("Upload is not open");
const received = new Set(row.receivedChunks);
for (let index = 0; index < chunkCountOf(row); index++) {
if (!received.has(index)) throw new MediaStateConflictError("Upload is missing chunks");
}
const updated = await tx.mediaUpload.update({
where: { id: row.id },
data: { status: "complete" },
});
await tx.mediaAsset.update({
where: { id: row.assetId },
data: {
storage: "uploaded",
storagePath: stored.storagePath,
contentHash: row.sha256,
sizeBytes: row.sizeBytes,
ingestState: "indexed",
ingestError: null,
},
});
return mapMediaUpload(updated);
});
}
async function failUpload(spaceId: string, uploadId: string): Promise<MediaUpload> {
const result = await prisma.mediaUpload.updateMany({
where: { id: uploadId, spaceId, status: "open" },
data: { status: "failed" },
});
if (result.count !== 1) {
await requireUpload(spaceId, uploadId);
throw new MediaStateConflictError("Upload is not open");
}
return uploadStatus(spaceId, uploadId);
}
return {
beginUpload,
uploadStatus,
getUploadRecord,
markChunkReceived,
completeUpload,
failUpload,
};
}

Some files were not shown because too many files have changed in this diff Show More