49431def83
Bots now announce every click and typing step before it runs, and the web app draws a cursor in the bot's colour that glides to that point over the live computer screen, ripples on clicks and shows typing. - contracts: ComputerPointerPayload on the existing thread.computer event - executor: computer_act and browser_act announce pointer steps (best effort); the page-browser helper reports element screen boxes, which are stripped before results reach the model - web: agentPointerFeed + AgentCursor over the side-panel preview and the full-window screen, mapped through the noVNC letterbox - in-app browser: native WebContentsView on desktop (own session, http(s) only, clamped bounds, back/forward, reload, DevTools, load errors) and a sandboxed iframe on the web; replaces the dead panel, the fake 50%/40% cursors and the unimplemented devServer bridge Claude-Session: https://claude.ai/code/session_01WiGooXtsKrZQmdszepRrNi Co-authored-by: Claude <noreply@anthropic.com>
39 lines
1.2 KiB
TypeScript
39 lines
1.2 KiB
TypeScript
import * as z from "zod";
|
|
|
|
export const ComputerPointerAction = z.enum([
|
|
"move",
|
|
"click",
|
|
"double_click",
|
|
"right_click",
|
|
"down",
|
|
"up",
|
|
"type",
|
|
"key",
|
|
"scroll",
|
|
]);
|
|
export type ComputerPointerAction = z.infer<typeof ComputerPointerAction>;
|
|
|
|
/** Screen pixels; absent for keyboard steps that act wherever the pointer already is. */
|
|
export const ComputerPointerStep = z.object({
|
|
action: ComputerPointerAction,
|
|
x: z.number().int().min(0).max(100_000).optional(),
|
|
y: z.number().int().min(0).max(100_000).optional(),
|
|
});
|
|
export type ComputerPointerStep = z.infer<typeof ComputerPointerStep>;
|
|
|
|
export const MAX_COMPUTER_POINTER_STEPS = 48;
|
|
|
|
/**
|
|
* Payload of a `thread.computer` event with `kind: "pointer"`: where a bot is about to point,
|
|
* click or type on its computer screen, so clients can draw its cursor over the live screen.
|
|
*/
|
|
export const ComputerPointerPayload = z.object({
|
|
kind: z.literal("pointer"),
|
|
screen: z.object({
|
|
width: z.number().int().positive().max(100_000),
|
|
height: z.number().int().positive().max(100_000),
|
|
}),
|
|
steps: z.array(ComputerPointerStep).min(1).max(MAX_COMPUTER_POINTER_STEPS),
|
|
});
|
|
export type ComputerPointerPayload = z.infer<typeof ComputerPointerPayload>;
|