Picks up ComfyUI generation, media tool dispatch and the streaming runner.
The video tools now share the image tools' schema helper, and both sets of
operations join the runtime's operation list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
edit_video and export_video now run as media jobs: both registrations join
the shared operation list the API and the worker build their runtime from,
name every artifact they read (source, appended clips, subtitle files) so
starting a job refuses artifacts the user cannot read, and their tool
schemas use the runtime's continue_after helper.
Container deployments get ffmpeg, ffprobe and the subtitle font in the
API/worker image. Tests cover the handlers against a fake job context:
cuts, blurred 9:16 reframes with subtitles, appended silent clips, audio
extraction, single frames, batch exports with posters and progress,
rotated phone video, clear input errors and cancellation. They skip where
ffmpeg is not installed; the planner and parser tests always run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now make images and short clips on the user's own GPU server and
rework images locally, through the media job runtime.
- generate_image: prompt, aspect preset or exact size, up to 4 images, optional
source image for image-to-image. Sizes target what the default models are
tuned for, and sources are fitted to that size before upload, so large
photos cannot exhaust GPU memory.
- generate_video: prompt, aspect, duration up to 10 s, optional start frame.
- edit_image: crop, rotate, flip, resize to a size or format (crop, pad or
blurred background), extend, convert, in that order, with sharp; an optional
prompt repaints the result on ComfyUI. Edits keep the source as parent.
- Registrations declare the input artifacts so jobs refuse ones the user
cannot read, and generation declares a fixed GPU memory floor (6 GB images,
12 GB video) that a custom workflow cannot lower.
- Tool definitions reuse the operation input schemas plus continue_after.
- Saving media settings rejects custom workflows the provider could not fill.
- Tests: sharp edits on generated fixtures, handlers against the ComfyUI
emulator, and a PostgreSQL suite running the operations through the media
runtime (card outputs, artifact versions, foreign artifacts, GPU wait).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media operations (generation, edits, transcription) need a runner that works
outside the agent turn, keeps a card in the conversation current and never
lets two processes put work on one GPU. Operations land in later commits; this
is the runtime they register with.
- Background job `media.run { jobId }` on a named `media` queue. Jobs carry an
optional queue: the Graphile publisher passes it as queueName and the
in-memory driver runs each queue one job at a time, in order.
- `createMediaRuntime` implements MediaRuntime (capabilities, start, get,
cancel) over a list of `MediaOperationRegistration`s and runs jobs:
start validates the input, creates the job and posts the bot's media card in
one transaction (idempotent per tool call through an operation key), then
enqueues it. The runner claims a fenced lease (an advisory lock keeps one
live lease at a time, so the GPU stays single even when the queue is
bypassed), renews it while polling for cancellation, writes progress and
outputs into the card, defers with backoff when the GPU is short of memory
or the operation asks to wait, bounds deferrals, crash restarts and run
time, always cleans the job's temp directory and always ends the card in a
terminal state. With continue_after the starting bot is woken in the same
transaction that ends the job (new run trigger `media`).
- The reconciler re-enqueues jobs whose enqueue was lost, whose runner died
or whose wait is over; a job stuck behind a dead worker's queue lock runs
outside the queue.
- Media backends are deployment-wide settings stored encrypted as the
`media` IntegrationProviderConfig row: `loadMediaSettings` and
`saveMediaSettings` with the same address policy as OpenAI-compatible
servers (no metadata hosts, public hosts only when allowed, HTTPS when a key
is sent), a saved key never follows the settings to another server, and the
view never returns keys. Contract in contracts/media-settings.ts.
- `createMediaOperations` is the one list both the API and the worker build
their runtime from (empty until the operations land), and media-tools.ts has
the shared `continue_after` schema helper for the tool definitions.
- Migration: lease, deferral, operation key and continueAfter columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
