Newer ffmpeg releases refuse to encode JPEG from limited-range video
frames, so poster frames failed on CI's ffmpeg while older versions
accepted the same command. The poster is converted to full-range
yuvj420p before encoding, which every supported ffmpeg version takes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The model saw edit_video and export_video with artifact_id next to
generate_image, generate_video, edit_image and transcribe with artifactId,
sourceArtifactId and negativePrompt. Mixed conventions in one tool list invite
wrong calls, so all media inputs now use snake_case like continue_after and the
other builtin tools: artifact_id, source_artifact_id, negative_prompt,
duration_seconds. Handlers, artifact checks, the media card's transcribe
action and the tests follow. Nothing has shipped, so stored job inputs need no
migration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A card that only says "Transcript" does not tell two transcriptions apart,
and it is English whatever the user's language.
- transcribe takes an optional title like the video operations, so a bot
names the card in the user's language; "Transcript" stays the default.
- Transcribe on a media card passes the source file name as the title.
- The E2E seeds the card through bootstrap as well as threads/get, since the
bot thread's first snapshot arrives with bootstrap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Generated clips (nothing binary committed) go through extractAudio: a video's
audio reaches the server as 16 kHz mono WAV, a 12-second recording is
uploaded in two ranges under a small limit, and a video without sound is
refused before anything is uploaded. Skipped where ffmpeg is missing, which
may include CI runners.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Edit cards showed English words such as "crop · flip · extend" to every user.
edit_image now takes an optional title written by the bot in the user's
language, used for the card and the file name like edit_video. Without one,
the card falls back to the repaint prompt or to language-neutral tokens only:
sizes, ratios, degrees, arrows and format names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in ffmpeg (probing, extractAudio) and the video operations. The
transcribe registration now gets the ffmpeg audio tools, so video is
transcribed through extractAudio with the demuxer forced from the MIME type,
and long recordings are converted and uploaded in ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A Postgres suite starts edit_video and export_video through the real
runtime and local artifact store: outputs become linked versions with
posters on the card, exports store one version per preset, appended clips
and subtitle files of another user are refused at start, and an
out-of-range cut fails the job with a readable error. It skips without
Postgres or ffmpeg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
- The GPU guard waited for free VRAM at the operation's floor even on a card
whose total memory is below it (an 8 GB card and a 12 GB video floor), so
such jobs deferred until their budget ran out. The requirement is now capped
at 90% of the card's total; a card with less than half the floor fails at
once with "The GPU does not have enough memory for this job."
- Tests pin that media tools reach a run only through the space's
capabilities and within the bot's allowlist now that the image tool
definitions exist.
- Type-only imports are top-level `import type` in the media runtime, tools,
RPCs and tests (AGENTS.md).
- The ComfyUI settings test gives saveMediaSettings a store with nothing saved,
since the save reads the previous settings before validating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The card's follow-ups now reach the media job RPCs, and it only offers what
the space can run.
- media.capabilities answers which operations the actor's space can run
(booleans only). The card asks once per minute per space, and only when it
shows finished video or audio, so other cards cost nothing.
- Export and Transcribe call media.jobs.start with the card's message, so the
same bot posts the new card; Cancel calls media.jobs.cancel. A refused
start shows the server's message under the card.
- New files use top-level import type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Picks up ComfyUI generation, media tool dispatch and the streaming runner.
The video tools now share the image tools' schema helper, and both sets of
operations join the runtime's operation list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in the media job RPCs, generic media tool dispatch and the image
operations; transcribe joins the operation list and the tool catalog next to
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A transcript is whatever was said in the recording, so the summary the bot
is woken with quotes it and says it is not instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Binary paths, the ffmpeg version, the probe's container name and a
configurable stall timeout had no caller; detection is cached per process,
so every helper looks the binaries up itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Covers what the unit tests fake: the job starts from validated input with
the "Transcript" card, runs against the transcription emulator, stores the
.srt and .vtt as artifacts next to the source and lists them on the card,
and an artifact of another user is refused before a job exists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
With the media runtime in place, transcription becomes an operation like the
others: bots call the transcribe tool, the job runs in the background and
the card shows the subtitle files.
- createTranscriptionOperations registers transcribe: available when a
transcription server is configured, refuses artifacts the user cannot
read before the job exists, and writes <name>.srt and <name>.vtt next to
the source (a .txt when the server returns no timings). With ffmpeg the
audio is converted to 16 kHz mono WAV and long media is transcribed in
upload-sized ranges on one timeline; without it audio is sent as stored
and video is refused with a clear message. A server that is down or busy
(unreachable, 429, 502, 503) defers the job instead of failing it, since
nothing has been stored yet. The bot gets the transcript, cut at a word.
- The transcribe tool definition uses the shared media schema helper and
continue_after; dispatch stays generic.
- The main bot's run creates Studio once a media server is configured,
right before the specialist directory is built.
- Dictation: when the user's voice provider cannot transcribe (or there is
none), /api/voice/transcribe uses the self-hosted transcription server,
and the voice status reports server dictation so the web app uses it
instead of the browser's speech recognition. A voice provider that
transcribes keeps being used as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
edit_video and export_video now run as media jobs: both registrations join
the shared operation list the API and the worker build their runtime from,
name every artifact they read (source, appended clips, subtitle files) so
starting a job refuses artifacts the user cannot read, and their tool
schemas use the runtime's continue_after helper.
Container deployments get ffmpeg, ffprobe and the subtitle font in the
API/worker image. Tests cover the handlers against a fake job context:
cuts, blurred 9:16 reframes with subtitles, appended silent clips, audio
extraction, single frames, batch exports with posters and progress,
rotated phone video, clear input errors and cancellation. They skip where
ffmpeg is not installed; the planner and parser tests always run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now make images and short clips on the user's own GPU server and
rework images locally, through the media job runtime.
- generate_image: prompt, aspect preset or exact size, up to 4 images, optional
source image for image-to-image. Sizes target what the default models are
tuned for, and sources are fitted to that size before upload, so large
photos cannot exhaust GPU memory.
- generate_video: prompt, aspect, duration up to 10 s, optional start frame.
- edit_image: crop, rotate, flip, resize to a size or format (crop, pad or
blurred background), extend, convert, in that order, with sharp; an optional
prompt repaints the result on ComfyUI. Edits keep the source as parent.
- Registrations declare the input artifacts so jobs refuse ones the user
cannot read, and generation declares a fixed GPU memory floor (6 GB images,
12 GB video) that a custom workflow cannot lower.
- Tool definitions reuse the operation input schemas plus continue_after.
- Saving media settings rejects custom workflows the provider could not fill.
- Tests: sharp edits on generated fixtures, handlers against the ComfyUI
emulator, and a PostgreSQL suite running the operations through the media
runtime (card outputs, artifact versions, foreign artifacts, GPU wait).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now cut, reframe and export videos for every common format without
a hosted service: ffmpeg and ffprobe from PATH do the work on the server.
- ffmpeg wrapper: PATH detection, ffprobe JSON probing (rotation, pixel
aspect, frame rate, audio, packet scan when the container has no
duration), runs with -progress parsing, abort and stall kills, an output
size cap, partial-output cleanup and short errors without temp paths.
Inputs go through the file protocol with the demuxer forced from the MIME
type, so a crafted upload cannot be probed as a playlist that reads other
files or URLs. extractAudio (16 kHz mono WAV) and renderPoster for
transcription and media cards.
- A pure planner turns edit steps (trim, keep, concat, reframe, speed,
volume, mute, subtitles, extract_audio, poster) into one filter_complex
pass: leading cuts seek the input, appended clips are normalized to the
frame, fps and 48 kHz stereo (silence where a clip has none), reframes
crop, pad or blur to the presets, speed chains atempo, subtitles burn in
from generated file names with a readable style that stays clear of the
controls on 9:16 frames. H.264/AAC MP4 with yuv420p and faststart;
location metadata is stripped.
- edit_video and export_video operations and tool definitions; the inputs
are shared contracts so the media card can start exports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Groundwork for the transcribe operation and for creating Studio when it is
first needed.
- Contracts: TranscribeInputSchema (artifact id, optional language code).
- transcribeChunks uploads audio chunks in order and stitches them into one
transcript on the original timeline; transcriptionChunkSeconds sizes
16 kHz mono WAV chunks below the server's upload limit; transcriptSummary
hands the bot the transcript, cut at a word.
- ensureStudioWhenMediaConfigured creates Studio once a media server
(generation or transcription) is configured. Local edits alone do not
count, because ffmpeg and image tools are present on most installs and a
specialist nobody asked for would appear for everyone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
