AGENTS.md asks for top-level `import type` instead of inline `type`
specifiers; this applies it to the import statements the media work added
or changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The main conversation lives in the home space, but the user can have another
space selected while it is on screen, and the card's actions sent that space:
media.jobs.start resolved the conversation in the selected space and failed,
and media.jobs.get/cancel only found jobs of the selected space.
The server now takes the space from the card itself. start finds the card's
message in one of the actor's own threads (the conversation named by botId or
groupId), requires the actor to still be a member of that thread's space, and
resolves the conversation and the source artifacts there. get and cancel find
the actor's job by id and require membership of the job's space.
media.capabilities takes an optional job id and answers for that job's space;
the card asks through its job and shares the answer per conversation. Every
other case is a 404, as before.
A Postgres test runs a card in the home-space main conversation while another
space is selected (start, get, capabilities, cancel) and checks that another
user and a user who left the card's space get 404 on every procedure.
This also completes the media settings section committed earlier as work in
progress (01eea83); its code was final, only that commit message said
otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The model saw edit_video and export_video with artifact_id next to
generate_image, generate_video, edit_image and transcribe with artifactId,
sourceArtifactId and negativePrompt. Mixed conventions in one tool list invite
wrong calls, so all media inputs now use snake_case like continue_after and the
other builtin tools: artifact_id, source_artifact_id, negative_prompt,
duration_seconds. Handlers, artifact checks, the media card's transcribe
action and the tests follow. Nothing has shipped, so stored job inputs need no
migration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The shell contrast test added on main measured the sidebar trading status,
which this branch removes; it now measures the rows that remain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Node's default 5-minute requestTimeout cut off a 500 MiB video upload on
any uplink slower than about 14 Mbit/s, so the video cap was unusable over
ordinary connections.
- The API server's requestTimeout is 30 minutes, enough for the largest
upload at about 2.5 Mbit/s. headersTimeout keeps its 60-second default.
- Each streamed upload aborts when the client sends no bytes for 60
seconds: the route answers 408, the partial file is deleted and no row
is written, so a slow-drip client cannot hold a connection for the full
request timeout. Time spent writing to disk does not count.
- The store and the upload guard no longer wait for a body to confirm its
cancellation, which a stalled or hostile client may never do.
- A guard test checks that every file the executor attaches records the
thread's group, which the render_plot chart fallback once dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Dictation falls back to the owner's transcription server when the voice
provider cannot transcribe, so that server is a voice recipient too: it is
listed for voice use (as "Transcription (<origin>)") whenever it would take
the recording, and not when the voice provider transcribes itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A card that only says "Transcript" does not tell two transcriptions apart,
and it is English whatever the user's language.
- transcribe takes an optional title like the video operations, so a bot
names the card in the user's language; "Transcript" stays the default.
- Transcribe on a media card passes the source file name as the title.
- The E2E seeds the card through bootstrap as well as threads/get, since the
bot thread's first snapshot arrives with bootstrap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The image E2E now runs in the main conversation, which is a group thread,
so the browser test covers group uploads and reads from the artifact route
as well as bot threads. Bot file attachments keep their run's operation id
for tracing, and the non-streaming attach cap is the inline cap directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Generated clips (nothing binary committed) go through extractAudio: a video's
audio reaches the server as 16 kHz mono WAV, a 12-second recording is
uploaded in two ranges under a small limit, and a video without sound is
refused before anything is uploaded. Skipped where ffmpeg is missing, which
may include CI runners.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Edit cards showed English words such as "crop · flip · extend" to every user.
edit_image now takes an optional title written by the bot in the user's
language, used for the card and the file name like edit_video. Without one,
the card falls back to the repaint prompt or to language-neutral tokens only:
sizes, ratios, degrees, arrows and format names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in ffmpeg (probing, extractAudio) and the video operations. The
transcribe registration now gets the ffmpeg audio tools, so video is
transcribed through extractAudio with the demuxer forced from the MIME type,
and long recordings are converted and uploaded in ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
- A handler that ignores its abort is left behind after the grace period;
a late rejection from it (or from the grace timer after the job settled)
was unobserved, and the worker exits on unhandled rejections. Both sides of
the race are now observed and the grace timer is cleared with the job.
- Ending a job removes its temp directory when no runner in this process
holds it (leftovers of a runner that died here), and reconciliation sweeps
directories of jobs that no longer run.
- An executor test runs a media tool end to end through applyTool: shown
only for operations the space can run, started with the run's scope and
effect key, continue_after split off, refused when not exposed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A Postgres suite starts edit_video and export_video through the real
runtime and local artifact store: outputs become linked versions with
posters on the card, exports store one version per preset, appended clips
and subtitle files of another user are refused at start, and an
out-of-range cut fails the job with a readable error. It skips without
Postgres or ffmpeg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
- The GPU guard waited for free VRAM at the operation's floor even on a card
whose total memory is below it (an 8 GB card and a 12 GB video floor), so
such jobs deferred until their budget ran out. The requirement is now capped
at 90% of the card's total; a card with less than half the floor fails at
once with "The GPU does not have enough memory for this job."
- Tests pin that media tools reach a run only through the space's
capabilities and within the bot's allowlist now that the image tool
definitions exist.
- Type-only imports are top-level `import type` in the media runtime, tools,
RPCs and tests (AGENTS.md).
- The ComfyUI settings test gives saveMediaSettings a store with nothing saved,
since the save reads the previous settings before validating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The card's follow-ups now reach the media job RPCs, and it only offers what
the space can run.
- media.capabilities answers which operations the actor's space can run
(booleans only). The card asks once per minute per space, and only when it
shows finished video or audio, so other cards cost nothing.
- Export and Transcribe call media.jobs.start with the card's message, so the
same bot posts the new card; Cancel calls media.jobs.cancel. A refused
start shows the server's message under the card.
- New files use top-level import type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Picks up ComfyUI generation, media tool dispatch and the streaming runner.
The video tools now share the image tools' schema helper, and both sets of
operations join the runtime's operation list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Uploads with a text/plain, text/csv or text/markdown body are CORS "simple"
requests that a browser sends cross-site without a preflight. The session
cookie's SameSite default already keeps it off such requests, but a
deployment that relaxes it would accept uploads from any page. The route now
applies the same origin rule as CORS before reading the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in the media job RPCs, generic media tool dispatch and the image
operations; transcribe joins the operation list and the tool catalog next to
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Agents only received the documents of a turn on their computer and could
attach workspace files up to 10 MiB, so they could neither cut a clip a user
sent nor post the video they rendered.
- Current-turn images, video and audio are copied into the bot's workspace
(Team Computers under bots/<botId>/) like documents. Remote computers move
files as base64 JSON, so anything above their 12 MiB transfer cap stays in
storage and the agent is told it by artifact id instead of failing the
turn; local computers copy up to the kind cap.
- attach_file accepts video and audio by extension and attaches up to the
kind cap when the store streams (still bounded by the transfer cap of a
remote computer), stored through the shared artifact helper with a real
SHA-256.
- The render_plot fallback that attaches a chart PNG now records the group,
so the chart opens in group conversations.
- Agent history names image artifacts by id, like video and audio, so a
later turn can refer to them.
- artifacts.get refuses files above 25 MiB: base64 reads hold the whole
file in memory, and larger media is served by the artifact route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A transcript is whatever was said in the recording, so the summary the bot
is woken with quotes it and says it is not instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Binary paths, the ffmpeg version, the probe's container name and a
configurable stall timeout had no caller; detection is cached per process,
so every helper looks the binaries up itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media tools and card actions send prompts, images, video and audio to the
deployment owner's ComfyUI and transcription servers. Like the embeddings
server behind built-in memory, those self-hosted endpoints are recipients in
the AI data-sharing disclosure: they appear with the model uses ("ComfyUI
(<origin>)", "Transcription (<origin>)", keyed by the full endpoint so a new
server asks again), and starting a card job asks for model consent on clients
that run the foreground check.
Also adds unit tests for the media settings store: URL normalization and
refusals, encryption, the key-free view, one decryption per saved version, key
retention per server and removing empty settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Covers what the unit tests fake: the job starts from validated input with
the "Transcript" card, runs against the transcription emulator, stores the
.srt and .vtt as artifacts next to the source and lists them on the card,
and an artifact of another user is refused before a job exists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
With the media runtime in place, transcription becomes an operation like the
others: bots call the transcribe tool, the job runs in the background and
the card shows the subtitle files.
- createTranscriptionOperations registers transcribe: available when a
transcription server is configured, refuses artifacts the user cannot
read before the job exists, and writes <name>.srt and <name>.vtt next to
the source (a .txt when the server returns no timings). With ffmpeg the
audio is converted to 16 kHz mono WAV and long media is transcribed in
upload-sized ranges on one timeline; without it audio is sent as stored
and video is refused with a clear message. A server that is down or busy
(unreachable, 429, 502, 503) defers the job instead of failing it, since
nothing has been stored yet. The bot gets the transcript, cut at a word.
- The transcribe tool definition uses the shared media schema helper and
continue_after; dispatch stays generic.
- The main bot's run creates Studio once a media server is configured,
right before the specialist directory is built.
- Dictation: when the user's voice provider cannot transcribe (or there is
none), /api/voice/transcribe uses the self-hosted transcription server,
and the voice status reports server dictation so the web app uses it
instead of the browser's speech recognition. A voice provider that
transcribes keeps being used as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots and users need to start and follow media jobs, and the deployment owner
needs to point the deployment at their GPU and transcription servers.
- Media tools are named like their operations. A run sees one only when the
space can run that operation (MediaRuntime.capabilities); the per-bot
allowlist still applies. applyTool hands every media tool to one
dispatcher (media-tools.ts) that splits off continue_after, starts the job
under the tool call's effect key (a replay returns the same job) and returns
at once; refusals (invalid input, unknown artifact, unavailable) go back to
the model as tool errors.
- media.jobs.start starts an export, transcription or image edit from a card:
the conversation is resolved like every thread procedure, the card's bot
(or the group's lead) posts the new card, and the runtime refuses source
artifacts outside the actor's space. media.jobs.get and media.jobs.cancel
only see the actor's own jobs.
- media.settings.get/update are deployment-owner only, never return keys and
answer invalid URLs with a 400 that says what is wrong.
- createApp accepts media operations as a test seam.
- Tests: a Postgres suite over the RPCs (owner-only settings, encryption,
key retention per server, URL policy, start/get/cancel isolation between
users, card jobs end to end) and unit tests for tool exposure and dispatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The image server is the deployment's, not necessarily the user's, and
repainting an edit only works when generation is available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
edit_video and export_video now run as media jobs: both registrations join
the shared operation list the API and the worker build their runtime from,
name every artifact they read (source, appended clips, subtitle files) so
starting a job refuses artifacts the user cannot read, and their tool
schemas use the runtime's continue_after helper.
Container deployments get ffmpeg, ffprobe and the subtitle font in the
API/worker image. Tests cover the handlers against a fake job context:
cuts, blurred 9:16 reframes with subtitles, appended silent clips, audio
extraction, single frames, batch exports with posters and progress,
rotated phone video, clear input errors and cancellation. They skip where
ffmpeg is not installed; the planner and parser tests always run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Queue, interrupt and prompt requests now go through one JSON helper, so the
cancel path reads like the rest of the client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now make images and short clips on the user's own GPU server and
rework images locally, through the media job runtime.
- generate_image: prompt, aspect preset or exact size, up to 4 images, optional
source image for image-to-image. Sizes target what the default models are
tuned for, and sources are fitted to that size before upload, so large
photos cannot exhaust GPU memory.
- generate_video: prompt, aspect, duration up to 10 s, optional start frame.
- edit_image: crop, rotate, flip, resize to a size or format (crop, pad or
blurred background), extend, convert, in that order, with sharp; an optional
prompt repaints the result on ComfyUI. Edits keep the source as parent.
- Registrations declare the input artifacts so jobs refuse ones the user
cannot read, and generation declares a fixed GPU memory floor (6 GB images,
12 GB video) that a custom workflow cannot lower.
- Tool definitions reuse the operation input schemas plus continue_after.
- Saving media settings rejects custom workflows the provider could not fill.
- Tests: sharp edits on generated fixtures, handlers against the ComfyUI
emulator, and a PostgreSQL suite running the operations through the media
runtime (card outputs, artifact versions, foreign artifacts, GPU wait).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media files can be hundreds of megabytes, so the runner no longer buffers them:
inputs are cloned from the local artifact file when the store has one and
streamed otherwise, outputs go through storeArtifactFromFile (per-kind size cap
and MIME check while streaming; an oversized result fails the job with a short
card error).
A Postgres suite runs the runtime with fake operations: success with outputs,
posters and parent versions; input, availability, artifact and thread checks
before a job exists; tool-call replay; failure with temp-dir cleanup; outputs
outside the temp dir and sources of another user; deferral with backoff and its
budget; the GPU memory guard; cancel of a queued job and of a job running in
another process; one job per lease and one job on the GPU at a time; crash
recovery and its restart limit; reconciliation inside and outside the queue;
the continue_after wake; a cleared conversation; the run time limit; and
capabilities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in artifact streaming and the media job runtime, settings and tool
helpers that the ComfyUI operations register with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in artifact streaming and the media job runtime that the video
operations register with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in streamed artifacts with artifactUrl (replacing the local stand-in)
and the media runtime, settings and operation registry the transcribe
operation plugs into.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now cut, reframe and export videos for every common format without
a hosted service: ffmpeg and ffprobe from PATH do the work on the server.
- ffmpeg wrapper: PATH detection, ffprobe JSON probing (rotation, pixel
aspect, frame rate, audio, packet scan when the container has no
duration), runs with -progress parsing, abort and stall kills, an output
size cap, partial-output cleanup and short errors without temp paths.
Inputs go through the file protocol with the demuxer forced from the MIME
type, so a crafted upload cannot be probed as a playlist that reads other
files or URLs. extractAudio (16 kHz mono WAV) and renderPoster for
transcription and media cards.
- A pure planner turns edit steps (trim, keep, concat, reframe, speed,
volume, mute, subtitles, extract_audio, poster) into one filter_complex
pass: leading cuts seek the input, appended clips are normalized to the
frame, fps and 48 kHz stereo (silence where a clip has none), reframes
crop, pad or blur to the presets, speed chains atempo, subtitles burn in
from generated file names with a readable style that stays clear of the
controls on 9:16 frames. H.264/AAC MP4 with yuv420p and faststart;
location metadata is stripped.
- edit_video and export_video operations and tool definitions; the inputs
are shared contracts so the media card can start exports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Groundwork for the transcribe operation and for creating Studio when it is
first needed.
- Contracts: TranscribeInputSchema (artifact id, optional language code).
- transcribeChunks uploads audio chunks in order and stitches them into one
transcript on the original timeline; transcriptionChunkSeconds sizes
16 kHz mono WAV chunks below the server's upload limit; transcriptSummary
hands the bot the transcript, cut at a word.
- ensureStudioWhenMediaConfigured creates Studio once a media server
(generation or transcription) is configured. Local edits alone do not
count, because ffmpeg and image tools are present on most installs and a
specialist nobody asked for would appear for everyone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media operations (generation, edits, transcription) need a runner that works
outside the agent turn, keeps a card in the conversation current and never
lets two processes put work on one GPU. Operations land in later commits; this
is the runtime they register with.
- Background job `media.run { jobId }` on a named `media` queue. Jobs carry an
optional queue: the Graphile publisher passes it as queueName and the
in-memory driver runs each queue one job at a time, in order.
- `createMediaRuntime` implements MediaRuntime (capabilities, start, get,
cancel) over a list of `MediaOperationRegistration`s and runs jobs:
start validates the input, creates the job and posts the bot's media card in
one transaction (idempotent per tool call through an operation key), then
enqueues it. The runner claims a fenced lease (an advisory lock keeps one
live lease at a time, so the GPU stays single even when the queue is
bypassed), renews it while polling for cancellation, writes progress and
outputs into the card, defers with backoff when the GPU is short of memory
or the operation asks to wait, bounds deferrals, crash restarts and run
time, always cleans the job's temp directory and always ends the card in a
terminal state. With continue_after the starting bot is woken in the same
transaction that ends the job (new run trigger `media`).
- The reconciler re-enqueues jobs whose enqueue was lost, whose runner died
or whose wait is over; a job stuck behind a dead worker's queue lock runs
outside the queue.
- Media backends are deployment-wide settings stored encrypted as the
`media` IntegrationProviderConfig row: `loadMediaSettings` and
`saveMediaSettings` with the same address policy as OpenAI-compatible
servers (no metadata hosts, public hosts only when allowed, HTTPS when a key
is sent), a saved key never follows the settings to another server, and the
view never returns keys. Contract in contracts/media-settings.ts.
- `createMediaOperations` is the one list both the API and the worker build
their runtime from (empty until the operations land), and media-tools.ts has
the shared `continue_after` schema helper for the tool definitions.
- Migration: lease, deferral, operation key and continueAfter columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
