The media pipeline moved image and file previews to the artifact route, so
they no longer take a bot or group target. The media card still passed one,
which no longer type-checks; only its follow-up actions need the target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Dictation falls back to the owner's transcription server when the voice
provider cannot transcribe, so that server is a voice recipient too: it is
listed for voice use (as "Transcription (<origin>)") whenever it would take
the recording, and not when the voice provider transcribes itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Uploaded videos carry no dimensions, so a vertical clip sat letterboxed in
a 16:9 frame. The player now takes the frame size from the file's metadata
when the block has none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The pipeline workstream's MediaPlayer takes artifactId, kind, name, poster
and size, without a MIME type; the card and the stand-in now use exactly
that, so its file replaces the stand-in without other changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A card that only says "Transcript" does not tell two transcriptions apart,
and it is English whatever the user's language.
- transcribe takes an optional title like the video operations, so a bot
names the card in the user's language; "Transcript" stays the default.
- Transcribe on a media card passes the source file name as the title.
- The E2E seeds the card through bootstrap as well as threads/get, since the
bot thread's first snapshot arrives with bootstrap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The image E2E now runs in the main conversation, which is a group thread,
so the browser test covers group uploads and reads from the artifact route
as well as bot threads. Bot file attachments keep their run's operation id
for tracing, and the non-streaming attach cap is the inline cap directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Generated clips (nothing binary committed) go through extractAudio: a video's
audio reaches the server as 16 kHz mono WAV, a 12-second recording is
uploaded in two ranges under a small limit, and a video without sound is
refused before anything is uploaded. Skipped where ffmpeg is missing, which
may include CI runners.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Edit cards showed English words such as "crop · flip · extend" to every user.
edit_image now takes an optional title written by the bot in the user's
language, used for the card and the file name like edit_video. Without one,
the card falls back to the repaint prompt or to language-neutral tokens only:
sizes, ratios, degrees, arrows and format names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in ffmpeg (probing, extractAudio) and the video operations. The
transcribe registration now gets the ffmpeg audio tools, so video is
transcribed through extractAudio with the demuxer forced from the MIME type,
and long recordings are converted and uploaded in ranges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Failed or cancelled jobs can still carry finished outputs, so the card asks
for the space's capabilities whenever a settled job shows video or audio.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A Postgres suite starts edit_video and export_video through the real
runtime and local artifact store: outputs become linked versions with
posters on the card, exports store one version per preset, appended clips
and subtitle files of another user are refused at start, and an
out-of-range cut fails the job with a readable error. It skips without
Postgres or ffmpeg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
- The GPU guard waited for free VRAM at the operation's floor even on a card
whose total memory is below it (an 8 GB card and a 12 GB video floor), so
such jobs deferred until their budget ran out. The requirement is now capped
at 90% of the card's total; a card with less than half the floor fails at
once with "The GPU does not have enough memory for this job."
- Tests pin that media tools reach a run only through the space's
capabilities and within the bot's allowlist now that the image tool
definitions exist.
- Type-only imports are top-level `import type` in the media runtime, tools,
RPCs and tests (AGENTS.md).
- The ComfyUI settings test gives saveMediaSettings a store with nothing saved,
since the save reads the previous settings before validating.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The card's follow-ups now reach the media job RPCs, and it only offers what
the space can run.
- media.capabilities answers which operations the actor's space can run
(booleans only). The card asks once per minute per space, and only when it
shows finished video or audio, so other cards cost nothing.
- Export and Transcribe call media.jobs.start with the card's message, so the
same bot posts the new card; Cancel calls media.jobs.cancel. A refused
start shows the server's message under the card.
- New files use top-level import type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Picks up ComfyUI generation, media tool dispatch and the streaming runner.
The video tools now share the image tools' schema helper, and both sets of
operations join the runtime's operation list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Uploads with a text/plain, text/csv or text/markdown body are CORS "simple"
requests that a browser sends cross-site without a preflight. The session
cookie's SameSite default already keeps it off such requests, but a
deployment that relaxes it would accept uploads from any page. The route now
applies the same origin rule as CORS before reading the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in the media job RPCs, generic media tool dispatch and the image
operations; transcribe joins the operation list and the tool catalog next to
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The media card's download link uses artifactUrl's download option so a video
or audio output is saved rather than opened inline, and the E2E seeds the
thread snapshot with fresh response headers so the patched body is read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Mobile has no media UI yet, so video, audio and media-card blocks rendered
as empty bubbles or fell out of previews.
- Thread text and reply previews describe video and audio files and the
media card's title and status.
- The bubble lists video, audio and a finished card's outputs as tappable
rows that open the file like other attachments.
- Opening video or audio downloads it from the artifact route to the cache
instead of a base64 RPC, so long recordings open without the server
holding them in memory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Agents only received the documents of a turn on their computer and could
attach workspace files up to 10 MiB, so they could neither cut a clip a user
sent nor post the video they rendered.
- Current-turn images, video and audio are copied into the bot's workspace
(Team Computers under bots/<botId>/) like documents. Remote computers move
files as base64 JSON, so anything above their 12 MiB transfer cap stays in
storage and the agent is told it by artifact id instead of failing the
turn; local computers copy up to the kind cap.
- attach_file accepts video and audio by extension and attaches up to the
kind cap when the store streams (still bounded by the transfer cap of a
remote computer), stored through the shared artifact helper with a real
SHA-256.
- The render_plot fallback that attaches a chart PNG now records the group,
so the chart opens in group conversations.
- Agent history names image artifacts by id, like video and audio, so a
later turn can refer to them.
- artifacts.get refuses files above 25 MiB: base64 reads hold the whole
file in memory, and larger media is served by the artifact route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
A transcript is whatever was said in the recording, so the summary the bot
is woken with quotes it and says it is not instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Binary paths, the ffmpeg version, the probe's container name and a
configurable stall timeout had no caller; detection is cached per process,
so every helper looks the binaries up itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media tools and card actions send prompts, images, video and audio to the
deployment owner's ComfyUI and transcription servers. Like the embeddings
server behind built-in memory, those self-hosted endpoints are recipients in
the AI data-sharing disclosure: they appear with the model uses ("ComfyUI
(<origin>)", "Transcription (<origin>)", keyed by the full endpoint so a new
server asks again), and starting a card job asks for model consent on clients
that run the foreground check.
Also adds unit tests for the media settings store: URL normalization and
refusals, encryption, the key-free view, one decryption per saved version, key
retention per server and removing empty settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Covers what the unit tests fake: the job starts from validated input with
the "Transcript" card, runs against the transcription emulator, stores the
.srt and .vtt as artifacts next to the source and lists them on the card,
and an artifact of another user is refused before a job exists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media sent in a conversation had nowhere to go: the composer capped every
file at 10 MiB and pushed it through a base64 RPC, video and audio blocks
did not render, and downloads decoded whole files in the page.
- Video and audio blocks play inline through MediaPlayer, which loads
nothing until the player is near the viewport, sizes the frame from the
block's dimensions (vertical clips stay within the column) and uses the
poster when one exists. Files the browser cannot play fall back to their
name with a download action.
- Images load straight from the artifact route; the lightbox gains a
download action.
- File cards, the Markdown preview and downloads read from the artifact
route instead of decoding base64.
- The composer accepts each kind up to its cap and names the computed
limit when it skips a file; video, audio and anything above the inline
cap upload as a raw stream, small files keep the inline RPC.
- MediaPlayer and artifactUrl are what the media card builds on.
Tests: MediaPlayer laziness, sizing and fallback, the upload helper, and an
E2E that uploads a tiny MP4 fixture (VP9, since Playwright's Chromium has
no H.264 decoder), plays it from the artifact route and captures
media-video-block, plus the image lightbox download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
With the media runtime in place, transcription becomes an operation like the
others: bots call the transcribe tool, the job runs in the background and
the card shows the subtitle files.
- createTranscriptionOperations registers transcribe: available when a
transcription server is configured, refuses artifacts the user cannot
read before the job exists, and writes <name>.srt and <name>.vtt next to
the source (a .txt when the server returns no timings). With ffmpeg the
audio is converted to 16 kHz mono WAV and long media is transcribed in
upload-sized ranges on one timeline; without it audio is sent as stored
and video is refused with a clear message. A server that is down or busy
(unreachable, 429, 502, 503) defers the job instead of failing it, since
nothing has been stored yet. The bot gets the transcript, cut at a word.
- The transcribe tool definition uses the shared media schema helper and
continue_after; dispatch stays generic.
- The main bot's run creates Studio once a media server is configured,
right before the specialist directory is built.
- Dictation: when the user's voice provider cannot transcribe (or there is
none), /api/voice/transcribe uses the self-hosted transcription server,
and the voice status reports server dictation so the web app uses it
instead of the browser's speech recognition. A voice provider that
transcribes keeps being used as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots and users need to start and follow media jobs, and the deployment owner
needs to point the deployment at their GPU and transcription servers.
- Media tools are named like their operations. A run sees one only when the
space can run that operation (MediaRuntime.capabilities); the per-bot
allowlist still applies. applyTool hands every media tool to one
dispatcher (media-tools.ts) that splits off continue_after, starts the job
under the tool call's effect key (a replay returns the same job) and returns
at once; refusals (invalid input, unknown artifact, unavailable) go back to
the model as tool errors.
- media.jobs.start starts an export, transcription or image edit from a card:
the conversation is resolved like every thread procedure, the card's bot
(or the group's lead) posts the new card, and the runtime refuses source
artifacts outside the actor's space. media.jobs.get and media.jobs.cancel
only see the actor's own jobs.
- media.settings.get/update are deployment-owner only, never return keys and
answer invalid URLs with a 400 that says what is wrong.
- createApp accepts media operations as a test seam.
- Tests: a Postgres suite over the RPCs (owner-only settings, encryption,
key retention per server, URL policy, start/get/cancel isolation between
users, card jobs end to end) and unit tests for tool exposure and dispatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
The image server is the deployment's, not necessarily the user's, and
repainting an edit only works when generation is available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
edit_video and export_video now run as media jobs: both registrations join
the shared operation list the API and the worker build their runtime from,
name every artifact they read (source, appended clips, subtitle files) so
starting a job refuses artifacts the user cannot read, and their tool
schemas use the runtime's continue_after helper.
Container deployments get ffmpeg, ffprobe and the subtitle font in the
API/worker image. Tests cover the handlers against a fake job context:
cuts, blurred 9:16 reframes with subtitles, appended silent clips, audio
extraction, single frames, batch exports with posters and progress,
rotated phone video, clear input errors and cancellation. They skip where
ffmpeg is not installed; the planner and parser tests always run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Queue, interrupt and prompt requests now go through one JSON helper, so the
cancel path reads like the rest of the client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now make images and short clips on the user's own GPU server and
rework images locally, through the media job runtime.
- generate_image: prompt, aspect preset or exact size, up to 4 images, optional
source image for image-to-image. Sizes target what the default models are
tuned for, and sources are fitted to that size before upload, so large
photos cannot exhaust GPU memory.
- generate_video: prompt, aspect, duration up to 10 s, optional start frame.
- edit_image: crop, rotate, flip, resize to a size or format (crop, pad or
blurred background), extend, convert, in that order, with sharp; an optional
prompt repaints the result on ComfyUI. Edits keep the source as parent.
- Registrations declare the input artifacts so jobs refuse ones the user
cannot read, and generation declares a fixed GPU memory floor (6 GB images,
12 GB video) that a custom workflow cannot lower.
- Tool definitions reuse the operation input schemas plus continue_after.
- Saving media settings rejects custom workflows the provider could not fill.
- Tests: sharp edits on generated fixtures, handlers against the ComfyUI
emulator, and a PostgreSQL suite running the operations through the media
runtime (card outputs, artifact versions, foreign artifacts, GPU wait).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media files can be hundreds of megabytes, so the runner no longer buffers them:
inputs are cloned from the local artifact file when the store has one and
streamed otherwise, outputs go through storeArtifactFromFile (per-kind size cap
and MIME check while streaming; an oversized result fails the job with a short
card error).
A Postgres suite runs the runtime with fake operations: success with outputs,
posters and parent versions; input, availability, artifact and thread checks
before a job exists; tool-call replay; failure with temp-dir cleanup; outputs
outside the temp dir and sources of another user; deferral with backoff and its
budget; the GPU memory guard; cancel of a queued job and of a job running in
another process; one job per lease and one job on the GPU at a time; crash
recovery and its restart limit; reconciliation inside and outside the queue;
the continue_after wake; a cleared conversation; the run time limit; and
capabilities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in artifact streaming and the media job runtime, settings and tool
helpers that the ComfyUI operations register with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in artifact streaming and the media job runtime that the video
operations register with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Brings in streamed artifacts with artifactUrl (replacing the local stand-in)
and the media runtime, settings and operation registry the transcribe
operation plugs into.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Bots can now cut, reframe and export videos for every common format without
a hosted service: ffmpeg and ffprobe from PATH do the work on the server.
- ffmpeg wrapper: PATH detection, ffprobe JSON probing (rotation, pixel
aspect, frame rate, audio, packet scan when the container has no
duration), runs with -progress parsing, abort and stall kills, an output
size cap, partial-output cleanup and short errors without temp paths.
Inputs go through the file protocol with the demuxer forced from the MIME
type, so a crafted upload cannot be probed as a playlist that reads other
files or URLs. extractAudio (16 kHz mono WAV) and renderPoster for
transcription and media cards.
- A pure planner turns edit steps (trim, keep, concat, reframe, speed,
volume, mute, subtitles, extract_audio, poster) into one filter_complex
pass: leading cuts seek the input, appended clips are normalized to the
frame, fps and 48 kHz stereo (silence where a clip has none), reframes
crop, pad or blur to the presets, speed chains atempo, subtitles burn in
from generated file names with a readable style that stays clear of the
controls on 9:16 frames. H.264/AAC MP4 with yuv420p and faststart;
location metadata is stripped.
- edit_video and export_video operations and tool definitions; the inputs
are shared contracts so the media card can start exports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Media jobs report progress and results through the media block, which the
web app did not render.
- MediaCard renders the block: operation icon, title, status (reusing the
task card's labels, plus "Waiting for GPU" for jobs queued behind the
GPU), a progress bar while active, the error when failed, and outputs:
images as thumbnails with the existing lightbox, video and audio in the
media player, subtitle and text files as file cards. Finished video and
audio get a download link, "Export" with presets labelled by aspect ratio
(the video's own shape left out) and "Transcribe", offered only when the
space can run them; running jobs get "Cancel".
- MessageView renders it with one branch.
- An E2E seeds the card on a thread snapshot with real uploaded artifacts.
- artifact-url.ts and MediaPlayer.tsx are minimal stand-ins for the
pipeline workstream's files and give way to them on merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
Groundwork for the transcribe operation and for creating Studio when it is
first needed.
- Contracts: TranscribeInputSchema (artifact id, optional language code).
- transcribeChunks uploads audio chunks in order and stitches them into one
transcript on the original timeline; transcriptionChunkSeconds sizes
16 kHz mono WAV chunks below the server's upload limit; transcriptSummary
hands the bot the transcript, cut at a word.
- ensureStudioWhenMediaConfigured creates Studio once a media server
(generation or transcription) is configured. Local edits alone do not
count, because ffmpeg and image tools are present on most installs and a
specialist nobody asked for would appear for everyone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRtGG9ARfqqhma3t85Ro2u
