cynarAI 63f96b6edd fix(ci): restore the web client and the failing product checks (#83)
* fix(ci): restore browser preview module and onboarding e2e

The web bundle imported a browser-preview store that was never added, so production builds and every Playwright signup failed before the shell rendered. Mobile check failed because Expo packages were a patch behind the SDK.

Restore the store, type the optional desktop bridge, and walk whichever onboarding step scripted CI actually opens.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): stop the blank web client and the send 500

The core barrel pulled git checkout into the browser, so node:child_process crashed the page before sign-up. Serve that module from a node export instead.

Add the missing run tool-call budget columns that threads/send writes. Guard an indexed shell flag for typecheck, and expect the files key the desktop preload already exposes.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): isolate pnpm setup and the takeover membership

The Team Computer takeover journey inserted a space member the
default-space trigger had already created, so shared Postgres rejected
the unique pair. Self-hosted jobs now keep a per-job pnpm home and
store, and artifact uploads no longer fail a job that already passed.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): clear typecheck, lint, and unit failures

Inbound channel rosters can mark a participant list as authoritative.
Answered channel asks stay off the group, and clearing a thread waits
until delegated returns land. Lint, unit, and web e2e contracts follow
the current shell, composer, and settings.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): align e2e with the shipped shell and onboarding

Web E2E still expected older focus buttons, nested project rows, and
settings that never opened MCP. Persist the signed-in space, list
non-default spaces under Projekte, and keep the quote control on screen.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): open group reply from a revealed hover rail

The first user row covered its Reply control until the rail accepted
pointer events. The group transcript now reveals that rail, then replies.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): stop the shell render loop and settle computer and mention UI

A working bot rewrote preview state on every render, the computer menu never stayed still, and the mention picker did not expose its expanded state or option name.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): stop idle probes hanging and accept a follow-up send

Background probes walked every process fd and timed out on the shared runner. Linux now tracks the marker with flock and /proc/locks. A new composer draft can send while an earlier send is still refreshing, and group members stay mentionable.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(adapters): detect background work when flock ids are padded

The idle probe compared /proc/locks with an unpadded device key, so majors
or minors under 16 never matched. It deleted the live marker and reported
idle while the command still held the lock. Probe by taking the lock, and
print the cancel key the way the kernel does.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(adapters): stabilize background-work probe tests on self-hosted CI

Clear stale marker files before each probe scenario and allow longer
polling so launch can acquire flock on busy persistent runners.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): do not block onboarding on spaces list

Render the first onboarding step from me and integration setup alone.
Resolve the optional skip-to-app redirect after that so Playwright
signups are not stuck on the loading shell when spaces.list is slow.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): restore green gates after second main rebase

Re-apply biome formatting on Shell/SortableGroup and strict TypeScript
assertions in computer-idle tests dropped when skipping the duplicate
conflict-resolution commit.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): wait for onboarding bootstrap before E2E proceeds

Expose data-onboarding-step on the onboarding shell and have Playwright
wait until loading finishes. Ignore ad-hoc screenshot scripts on
self-hosted runners so they do not fail lint.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): fall back when onboarding bootstrap stalls

Race me/integration setup against a timeout and promote loading to the
emoji step after 20s so Playwright is not blocked on a hung RPC.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): render onboarding immediately for E2E signups

Start on the emoji step instead of a loading shell that waits on RPC.
Bootstrap can still redirect once me and integration setup resolve.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(ci): stabilize web e2e on self-hosted runners

Per-job pnpm stores broke node_modules symlinks when setup-node restored
cached installs, so Vite could not resolve react during dependency
optimization. Keep a runner-scoped store, clear stale Vite caches, and pin
Playwright to one worker in CI to reduce onboarding RPC contention.

Add Vite resolve aliases for react/react-dom, wait for an interactive
onboarding step in E2E helpers, and include the React hook smoke spec from
main.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): resolve React aliases through pnpm package layout

Root node_modules/react does not exist under pnpm, so Vite aliases pointed
at missing paths and broke production builds and the Playwright webServer.
Resolve react and react-dom from the web package and map jsx-runtime to the
actual entry files.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(core): type the lazy child_process spawn cache

Main's lazy require left spawn typed as null after assignment, which broke
Typecheck on the rebased branch.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(db): mock git-checkout export path in unit tests

Vitest resolves vi.mock against package exports; the .js suffix is not
declared for @rakazo/core/git-checkout.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): stabilize E2E after main sidebar and onboarding changes

Use first() for duplicate transcript lines, wait for interactive onboarding
before model-required assertions, and allow project sidebar to settle after
create. Extend the Playwright job timeout so the single-worker suite can finish.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): restore Projekte sidebar and integrations MCP entry

Use the default space for Unterhaltungen/Projekte separation instead of the
active space header, persist the bootstrapped space in localStorage, wire
SettingsOverlay to MCP management, and compare hover tones on inactive rows.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): satisfy biome format on homeSpaceId line

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): align sidebar E2E with nested Projekte shell layout

Scope Unterhaltungen assertions to conversation rows only, target project-space
Chief actions under .sidebar-projekte, and stabilize hover-tone polling.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix: satisfy lint after main merge (format and unused var)

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): show project-space 1:1 bots in Projekte sidebar

Include lone bots in projectSidebarGroups, render bot rows in Projekte,
and stabilize E2E for nested sidebar layout and duplicate group entries.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): disambiguate duplicate group sidebar buttons in E2E

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): align spaces and group-chats E2E with Projekte sidebar IA

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): dedupe Projekte sidebar rows and stabilize E2E specs

- Deduplicate project space and group-chat entries in projectSidebarGroups
- Scope group-chats/spaces E2E clicks to first Projekte match after IA split
- Wait for shell ready before mention-picker keyboard test; restore main locators
- Fix PrivacySettingsOverlay lint and align biome schema with CLI 2.5.13

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): restrict project drag handle and stabilize space E2E

- Move SortableGroup drag listeners to ProjectDragGrip only so sidebar clicks work
- Scope space delete actions to project-space rows; wait for menu in E2E
- Switch group settings E2E via direct navigation after unsaved draft check
- Merge main (DataTrustPanel) and fix Shell import order for CI lint

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): stabilize message hover Escape assertion in E2E

Assert the More menu closes and the hover rail stays visible instead of
requiring focus restoration, which is flaky across Chromium builds.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* chore: format apps/api/src/app.ts after main merge

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

* fix(web): stabilize team-computer E2E send after bot switch

Wait for shell ready and use the Send button instead of Enter after
switching bots and closing the computer overlay. The prior run timed
out on threads/send with only an apps/api format commit in between.

Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: cynarAI <cynarAI@users.noreply.github.com>
2026-09-23 22:09:49 +02:00

1308 lines
42 KiB
TypeScript

import type { RealtimeFanout } from "@rakazo/adapter-kit";
import {
type BotSecretDestination,
type MessageBlock,
MessageBlock as MessageBlockSchema,
type ProductEvent,
} from "@rakazo/contracts";
import {
blocksToAgentHistoryText,
isApprovalAskBlock,
isSecretAskBlock,
messagingChannelId,
sanitizeJsonValue,
} from "@rakazo/core";
import { getLogger } from "@rakazo/logging";
import { cancelRunsInTransaction } from "./cancel-runs.js";
import type { Prisma, PrismaClient } from "./client.js";
import { expireComputerExecutionLeases } from "./computers.js";
import {
assertRunCanWriteHistory,
createThreadMessageInTransaction,
RunHistoryWriteError,
} from "./messages.js";
import { ClearThreadPendingDelegatedReturnsError } from "./scope.js";
import { withTransactionRetry } from "./transaction-retry.js";
const EVENT_BATCH_SIZE = 200;
const PUSH_CATCH_UP_MS = 30_000;
const POLL_ONLY_CATCH_UP_MS = 400;
export interface AppendEventInput {
spaceId: string;
threadId: string;
botId: string;
type: ProductEvent["type"];
payload: Record<string, unknown>;
runId?: string;
}
export interface ThreadEvents {
answerRunInput(input: AnswerRunInput): Promise<boolean>;
append(input: AppendEventInput): Promise<ProductEvent>;
claimSteering(input: ClaimSteeringInput): Promise<ClaimedSteeringMessage[]>;
clearThread(input: ClearThreadInput): Promise<ClearThreadResult>;
finalizeComputerControlRelease(
input: FinalizeComputerControlReleaseInput,
): Promise<FinalizeComputerControlReleaseResult | false>;
finalizeRun(input: FinalizeRunInput): Promise<FinalizeRunResult | false>;
notify(threadId: string, seq: number): Promise<void>;
pauseRunForInput(input: PauseRunForInput): Promise<boolean>;
pauseRunForTakeover(input: PauseRunForTakeover): Promise<boolean>;
sendUserMessage(input: SendUserMessageInput): Promise<SendUserMessageResult>;
follow(threadId: string, cursor: number, signal?: AbortSignal): AsyncGenerator<ProductEvent>;
}
export interface ClearThreadInput {
spaceId: string;
threadId: string;
/** Event author and the bot-scoped target for one-to-one chats. */
botId: string;
groupId?: string;
}
export interface ClearThreadResult {
event: ProductEvent;
cancelledRunIds: string[];
historyCompactionGeneration: number;
}
export interface FinalizeComputerControlReleaseInput {
spaceId: string;
computerId: string;
botId: string;
runId: string | null;
leaseId: string;
holder: "bot" | "none";
reason: "done" | "expired" | "released" | "skipped";
}
export interface FinalizeComputerControlReleaseResult {
runId: string | null;
}
export interface ClaimSteeringInput {
threadId: string;
botId: string;
runId: string;
leaseOwner: string;
leaseFence: number;
seenIds: string[];
}
export interface ClaimedSteeringMessage {
id: string;
messageId: string;
text: string;
blocks: MessageBlock[];
}
export interface FinalizeRunResult {
continuationRunId: string | null;
}
interface FinalizeRunBase {
spaceId: string;
threadId: string;
botId: string;
runId: string;
taskId: string;
attemptId: string;
leaseOwner: string;
leaseFence: number;
}
export type FinalizeRunInput = FinalizeRunBase &
(
| {
outcome: "completed";
blocks: MessageBlock[];
markUnread?: boolean;
}
| { outcome: "failed"; error: string }
);
export interface PauseRunForInput {
spaceId: string;
threadId: string;
botId: string;
runId: string;
attemptId: string;
leaseOwner: string;
leaseFence: number;
blocks: MessageBlock[];
/**
* Unredacted choice actions for resume. Persisted only on the run checkpoint
* (not in message blocks) so clients still see redacted labels.
*/
offeredActions?: Array<{ id: string; label: string }>;
}
const CHOICE_ASK_CHECKPOINT_KIND = "choice_ask_v1";
function choiceAskCheckpoint(actions: Array<{ id: string; label: string }>): string {
return JSON.stringify({ kind: CHOICE_ASK_CHECKPOINT_KIND, actions });
}
function resumeChoiceLabel(
selected: { id: string; label: string },
checkpoint: string | null | undefined,
): string {
if (!checkpoint) return selected.label;
try {
const parsed = JSON.parse(checkpoint) as {
kind?: string;
actions?: Array<{ id?: unknown; label?: unknown }>;
};
if (parsed.kind !== CHOICE_ASK_CHECKPOINT_KIND || !Array.isArray(parsed.actions)) {
return selected.label;
}
const offered = parsed.actions.find((action) => action.id === selected.id);
return typeof offered?.label === "string" && offered.label.length > 0
? offered.label
: selected.label;
} catch {
return selected.label;
}
}
export interface PauseRunForTakeover {
spaceId: string;
threadId: string;
botId: string;
runId: string;
attemptId: string;
leaseOwner: string;
leaseFence: number;
reason: string;
/** Computer that should expose the pending takeover to the UI via controlRunId. */
computerId: string;
}
export interface AnswerRunInput {
spaceId: string;
threadId: string;
runId: string;
messageId: string;
answeredByUserId: string;
answer: string;
}
export interface SendUserMessageInput {
spaceId: string;
threadId: string;
botId: string;
userId: string;
blocks: MessageBlock[];
prompt: string;
trigger: "user" | "follow_up" | "webhook" | "messaging";
clientNonce?: string;
linkMessageToRun?: boolean;
/** When false, persist the user message without starting a run (team-chat transcript). */
createRun?: boolean;
/** When true, start a new run even if the bot is already busy (team-chat delivery). */
allowParallelRun?: boolean;
}
export interface SendUserMessageResult {
messageId: string;
seq: number;
taskId: string | null;
runId: string | null;
}
export interface RunSecretWriter {
store(input: {
botId: string;
credential?: BotSecretDestination;
runId: string;
userId: string;
spaceId: string;
plaintext: string;
tx: Prisma.TransactionClient;
}): Promise<void>;
}
export function createThreadEvents(
prisma: PrismaClient,
realtime?: RealtimeFanout,
options: { catchUpMs?: number; runSecretWriter?: RunSecretWriter } = {},
): ThreadEvents {
return {
answerRunInput: (input) => answerRunInput(prisma, input, realtime, options.runSecretWriter),
append: (input) => appendEvent(prisma, input, realtime),
claimSteering: (input) => claimSteering(prisma, input),
clearThread: (input) => clearThread(prisma, input, realtime),
finalizeComputerControlRelease: (input) =>
finalizeComputerControlRelease(prisma, input, realtime),
finalizeRun: (input) => finalizeRun(prisma, input, realtime),
notify: (threadId, seq) => notifyRealtime(realtime, threadId, seq),
pauseRunForInput: (input) => pauseRunForInput(prisma, input, realtime),
pauseRunForTakeover: (input) => pauseRunForTakeover(prisma, input, realtime),
sendUserMessage: (input) => sendUserMessage(prisma, input, realtime),
follow: (threadId, cursor, signal) =>
followThreadEvents(prisma, threadId, cursor, realtime, signal, options.catchUpMs),
};
}
export async function clearThread(
prisma: PrismaClient,
input: ClearThreadInput,
realtime?: RealtimeFanout,
): Promise<ClearThreadResult> {
const committed = await prisma.$transaction(async (tx: Prisma.TransactionClient) => {
const thread = await tx.thread.update({
where: {
id: input.threadId,
spaceId: input.spaceId,
...(input.groupId ? { groupId: input.groupId } : { botId: input.botId }),
},
data: { unread: false },
select: { nextMessageSeq: true, historyCompactionGeneration: true },
});
const pendingDelegatedReturns = await tx.run.count({
where: {
spaceId: input.spaceId,
threadId: input.threadId,
...(input.groupId ? {} : { botId: input.botId }),
trigger: "bot_message",
status: { in: ["completed", "failed"] },
botOutcomeReturnedAt: null,
},
});
if (pendingDelegatedReturns > 0) {
throw new ClearThreadPendingDelegatedReturnsError();
}
const activeRuns = await tx.run.findMany({
where: {
spaceId: input.spaceId,
threadId: input.threadId,
...(input.groupId ? {} : { botId: input.botId }),
status: { in: ["queued", "leased", "running", "waiting_input", "waiting_takeover"] },
},
select: { id: true, taskId: true },
});
const now = new Date();
const runIds = activeRuns.map((run) => run.id);
await cancelRunsInTransaction(tx, activeRuns, now);
// Expire as tombstones so a still-open provider screen claim cannot reset fencing to 1.
await expireComputerExecutionLeases(tx, { runId: { in: runIds } });
await tx.computer.updateMany({
where: { executionRunId: { in: runIds } },
data: {
executionRunId: null,
executionBotId: null,
executionLeaseExpiresAt: null,
},
});
await tx.message.deleteMany({ where: { threadId: input.threadId } });
await tx.event.deleteMany({ where: { threadId: input.threadId } });
if (thread.nextMessageSeq > 0) {
// nextMessageSeq is not reset, so mark every deleted message as already compacted.
// Leaving the cursor behind would let compaction re-summarize deleted history (or, reset
// to null, immediately re-fire on the fresh conversation).
await tx.thread.update({
where: { id: input.threadId },
data: {
historyCompactedUpToSeq: thread.nextMessageSeq - 1,
historyCompactionSummary: null,
historyCompactionGeneration: { increment: 1 },
},
});
} else {
await tx.thread.update({
where: { id: input.threadId },
data: {
historyCompactionSummary: null,
historyCompactionGeneration: { increment: 1 },
},
});
}
if (input.groupId) {
await tx.chatGroup.update({ where: { id: input.groupId }, data: { updatedAt: now } });
} else {
await tx.bot.update({
where: { id: input.botId, spaceId: input.spaceId },
data: { updatedAt: now },
});
}
const event = await appendEventInTransaction(tx, {
...input,
type: "thread.cleared",
payload: {},
});
return {
event,
cancelledRunIds: runIds,
historyCompactionGeneration: thread.historyCompactionGeneration,
};
});
await notifyRealtime(realtime, committed.event.threadId, committed.event.seq);
return {
event: mapProductEvent(committed.event),
cancelledRunIds: committed.cancelledRunIds,
historyCompactionGeneration: committed.historyCompactionGeneration,
};
}
export async function sendUserMessage(
prisma: PrismaClient,
input: SendUserMessageInput,
realtime?: RealtimeFanout,
): Promise<SendUserMessageResult> {
const replay = async (): Promise<SendUserMessageResult | null> => {
if (!input.clientNonce) return null;
const message = await prisma.message.findUnique({
where: {
threadId_clientNonce: {
threadId: input.threadId,
clientNonce: input.clientNonce,
},
},
include: { sourceRuns: { orderBy: { createdAt: "asc" }, take: 1 } },
});
if (!message) return null;
const created = message.sourceRuns[0];
const run =
created ??
(message.runId ? await prisma.run.findUnique({ where: { id: message.runId } }) : null);
return {
messageId: message.id,
seq: message.seq,
taskId: created?.taskId ?? null,
runId: run?.id ?? null,
};
};
const existing = await replay();
if (existing) return existing;
const commit = () =>
prisma.$transaction(async (tx: Prisma.TransactionClient) => {
// Message first: its thread-row lock serializes the whole send against clearThread, so a
// concurrent clear either sees the committed run and cancels it, or strictly precedes this
// transaction. Created in separate transactions, the run could land inside the clear's
// window and later repopulate the cleared conversation, and a clear could strand the
// message without its event.
const message = await createThreadMessageInTransaction(tx, {
threadId: input.threadId,
role: "user",
blocks: input.blocks,
clientNonce: input.clientNonce,
});
const createRun = input.createRun !== false;
const busy =
createRun && !input.allowParallelRun
? await tx.run.findFirst({
where: {
threadId: input.threadId,
botId: input.botId,
status: {
in: ["running", "queued", "leased", "waiting_input", "waiting_takeover"],
},
},
select: { id: true, taskId: true },
})
: null;
let task = null;
let run = null;
if (createRun && !busy) {
task = await tx.task.create({
data: {
spaceId: input.spaceId,
botId: input.botId,
threadId: input.threadId,
userId: input.userId,
prompt: input.prompt,
status: "queued",
},
});
run = await tx.run.create({
data: {
spaceId: input.spaceId,
botId: input.botId,
threadId: input.threadId,
taskId: task.id,
userId: input.userId,
status: "queued",
trigger: input.trigger,
clientNonce: input.clientNonce ? `send:${message.id}` : undefined,
sourceMessageId: message.id,
},
});
if (input.linkMessageToRun) {
await tx.message.update({ where: { id: message.id }, data: { runId: run.id } });
}
} else if (createRun && busy) {
await tx.steeringMessage.create({
data: {
messageId: message.id,
botId: input.botId,
userId: input.userId,
runId: busy.id,
},
});
await tx.message.update({ where: { id: message.id }, data: { runId: busy.id } });
}
const event = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: "thread.message.created",
runId: run?.id ?? busy?.id,
payload: { messageId: message.id, role: "user", blocks: input.blocks },
});
return { message, task, run, busy, event };
});
const committed = await commit().catch(async (error) => {
const winner = await replay();
if (winner) return { replay: winner } as const;
throw error;
});
if ("replay" in committed) return committed.replay;
await notifyRealtime(realtime, input.threadId, committed.event.seq).catch((error) => {
// The event is durable; subscribers recover it from their persisted cursor.
getLogger().error("user message realtime notification", error);
});
return {
messageId: committed.message.id,
seq: committed.message.seq,
taskId: committed.task?.id ?? null,
runId: committed.run?.id ?? committed.busy?.id ?? null,
};
}
export async function claimSteering(
prisma: PrismaClient,
input: ClaimSteeringInput,
): Promise<ClaimedSteeringMessage[]> {
return prisma.$transaction(async (tx: Prisma.TransactionClient) => {
await tx.$queryRaw`SELECT id FROM threads WHERE id = ${input.threadId} FOR UPDATE`;
const run = await tx.run.findFirst({
where: {
id: input.runId,
threadId: input.threadId,
botId: input.botId,
status: "running",
leaseOwner: input.leaseOwner,
leaseFence: input.leaseFence,
},
select: { id: true, trigger: true, sourceMessage: { select: { blocks: true } } },
});
if (!run) return [];
const channelId =
run.trigger === "messaging"
? messagingChannelId(run.sourceMessage?.blocks as MessageBlock[] | undefined)
: undefined;
const steering = await tx.steeringMessage.findMany({
where: {
botId: input.botId,
claimedAt: null,
id: input.seenIds.length ? { notIn: input.seenIds } : undefined,
OR: [{ runId: null }, { runId: input.runId }],
message: {
threadId: input.threadId,
// Private follow-ups remain unclaimed for the existing private continuation.
...(channelId
? { blocks: { array_contains: [{ kind: "channel_message", channelId }] } }
: {}),
},
},
include: { message: { select: { blocks: true, seq: true } } },
orderBy: [{ message: { seq: "asc" } }, { id: "asc" }],
});
if (steering.length === 0) return [];
const claimed = await tx.steeringMessage.updateMany({
where: { id: { in: steering.map((item) => item.id) }, claimedAt: null },
data: { runId: input.runId, claimedAt: new Date() },
});
if (claimed.count === 0) return [];
return steering.map((item) => ({
id: item.id,
messageId: item.messageId,
text: blocksToAgentHistoryText(item.message.blocks as MessageBlock[]),
blocks: item.message.blocks as MessageBlock[],
}));
});
}
export async function answerRunInput(
prisma: PrismaClient,
input: AnswerRunInput,
realtime?: RealtimeFanout,
runSecretWriter?: RunSecretWriter,
): Promise<boolean> {
const committed = await prisma.$transaction(async (tx: Prisma.TransactionClient) => {
// Thread row first, then run rows — the same order as clearThread and finalizeRun, so a
// concurrent clear cannot deadlock against this transaction.
await tx.$queryRaw`SELECT id FROM threads WHERE id = ${input.threadId} FOR UPDATE`;
const run = await tx.run.findFirst({
where: {
id: input.runId,
spaceId: input.spaceId,
threadId: input.threadId,
status: "waiting_input",
},
select: { botId: true, userId: true, checkpoint: true },
});
if (!run) return null;
const message = await tx.message.findFirst({
where: {
id: input.messageId,
threadId: input.threadId,
runId: input.runId,
role: "bot",
},
});
const parsed = MessageBlockSchema.array().safeParse(message?.blocks);
if (!message || !parsed.success) return null;
const pendingAsk = parsed.data.find(
(block) => block.kind === "ask" && block.status !== "answered",
);
if (pendingAsk?.kind !== "ask") return null;
const approvalAsk = isApprovalAskBlock(pendingAsk);
const secretAsk = isSecretAskBlock(pendingAsk);
const choiceAsk = !approvalAsk && !secretAsk && Boolean(pendingAsk.actions?.length);
const selectedChoice = choiceAsk
? pendingAsk.actions?.find((action) => action.id === input.answer)
: undefined;
if (choiceAsk && !selectedChoice) return null;
if (secretAsk && !runSecretWriter) return null;
if (secretAsk && run.userId !== input.answeredByUserId) return null;
let approvalEffect: { id: string; kind: string } | null = null;
let approvalUserId: string | null = null;
if (approvalAsk) {
if (run.userId !== input.answeredByUserId) return null;
if (!pendingAsk.actions?.some((action) => action.id === input.answer)) return null;
approvalEffect = await tx.externalEffect.findFirst({
where: {
id: pendingAsk.approvalEffectId,
spaceId: input.spaceId,
runId: input.runId,
status: "intended",
},
});
if (!approvalEffect) return null;
if (input.answer === "always") {
approvalUserId = input.answeredByUserId;
}
}
const queued = await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
threadId: input.threadId,
status: "waiting_input",
},
data: {
status: "queued",
...(choiceAsk ? { checkpoint: null } : {}),
},
});
if (queued.count !== 1) return null;
if (approvalAsk) {
const allowed = input.answer === "allow" || input.answer === "always";
await tx.externalEffect.update({
where: { id: approvalEffect!.id },
data: { status: allowed ? "approved" : "denied" },
});
if (input.answer === "always") {
await tx.actionApprovalRule.upsert({
where: {
spaceId_createdByUserId_effect_matchKind_matchValue: {
spaceId: input.spaceId,
createdByUserId: approvalUserId!,
effect: "always_allow",
matchKind: "tool",
matchValue: approvalEffect!.kind,
},
},
create: {
spaceId: input.spaceId,
createdByUserId: approvalUserId!,
effect: "always_allow",
matchKind: "tool",
matchValue: approvalEffect!.kind,
},
update: {},
});
}
} else if (secretAsk) {
await runSecretWriter!.store({
botId: run.botId,
credential: pendingAsk.credential,
runId: input.runId,
userId: run.userId,
spaceId: input.spaceId,
plaintext: input.answer,
tx,
});
await tx.externalEffect.updateMany({
where: {
runId: input.runId,
spaceId: input.spaceId,
kind: "request_secret",
status: "intended",
},
data: {
status: "approved",
...(pendingAsk.credential ? { result: { credentialSaved: pendingAsk.credential } } : {}),
},
});
} else {
const resumeLabel = selectedChoice
? resumeChoiceLabel(selectedChoice, run.checkpoint)
: undefined;
const task = await tx.task.updateMany({
where: { runs: { some: { id: input.runId } } },
data: {
prompt: selectedChoice
? `Selected choice ${selectedChoice.id}: ${resumeLabel}`
: input.answer,
},
});
if (task.count !== 1) throw new Error("Run task was not available to answer");
}
const blocks = parsed.data.map((block) =>
block === pendingAsk
? {
...block,
status: "answered" as const,
answer: secretAsk ? "" : input.answer,
}
: block,
);
await tx.message.update({ where: { id: message.id }, data: { blocks } });
const updated = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: run.botId,
type: "thread.message.updated",
runId: input.runId,
payload: { messageId: message.id, role: "bot", blocks },
});
return { threadId: updated.threadId, seq: updated.seq };
});
if (!committed) return false;
await notifyRealtime(realtime, committed.threadId, committed.seq);
return true;
}
export async function pauseRunForInput(
prisma: PrismaClient,
input: PauseRunForInput,
realtime?: RealtimeFanout,
): Promise<boolean> {
const committed = await prisma.$transaction(async (tx: Prisma.TransactionClient) => {
// Thread row first, then run rows — the same order as clearThread and finalizeRun, so a
// concurrent clear cannot deadlock against this transaction.
await tx.$queryRaw`SELECT id FROM threads WHERE id = ${input.threadId} FOR UPDATE`;
const paused = await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
status: "running",
leaseOwner: input.leaseOwner,
leaseFence: input.leaseFence,
},
data: {
status: "waiting_input",
leaseOwner: null,
leaseExpiresAt: null,
...(input.offeredActions?.length
? { checkpoint: choiceAskCheckpoint(input.offeredActions) }
: {}),
},
});
if (paused.count !== 1) return null;
const attempt = await tx.attempt.updateMany({
where: {
id: input.attemptId,
runId: input.runId,
fence: input.leaseFence,
status: "running",
},
data: { status: "waiting_input", finishedAt: new Date() },
});
if (attempt.count !== 1) throw new Error("Active run attempt was not available to pause");
const message = await createThreadMessageInTransaction(tx, {
threadId: input.threadId,
role: "bot",
blocks: input.blocks,
botId: input.botId,
runId: input.runId,
});
await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: "thread.message.created",
runId: input.runId,
payload: { messageId: message.id, role: "bot", blocks: input.blocks },
});
const waitingEvent = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: "run.waiting_input",
runId: input.runId,
payload: {},
});
await tx.event.deleteMany({ where: { runId: input.runId, type: "thread.progress" } });
return { threadId: waitingEvent.threadId, seq: waitingEvent.seq };
});
if (!committed) return false;
await notifyRealtime(realtime, committed.threadId, committed.seq);
return true;
}
export async function pauseRunForTakeover(
prisma: PrismaClient,
input: PauseRunForTakeover,
realtime?: RealtimeFanout,
): Promise<boolean> {
const committed = await prisma.$transaction(async (tx: Prisma.TransactionClient) => {
await tx.$queryRaw`SELECT id FROM threads WHERE id = ${input.threadId} FOR UPDATE`;
const paused = await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
status: "running",
leaseOwner: input.leaseOwner,
leaseFence: input.leaseFence,
},
data: {
status: "waiting_takeover",
leaseOwner: null,
leaseExpiresAt: null,
checkpoint: null,
},
});
if (paused.count !== 1) return null;
const attempt = await tx.attempt.updateMany({
where: {
id: input.attemptId,
runId: input.runId,
fence: input.leaseFence,
status: "running",
},
data: { status: "waiting_takeover", finishedAt: new Date() },
});
if (attempt.count !== 1) throw new Error("Active run attempt was not available to pause");
const now = new Date();
// Serialize with concurrent computer/takeover grants that mutate the same row.
await tx.$queryRaw`
SELECT id FROM computers WHERE id = ${input.computerId} AND "spaceId" = ${input.spaceId} FOR UPDATE`;
const computer = await tx.computer.findFirst({
where: { id: input.computerId, spaceId: input.spaceId },
select: {
controlHolder: true,
controlBotId: true,
controlLeaseId: true,
controlLeaseExpiresAt: true,
},
});
const activeUserLease = Boolean(
computer?.controlHolder === "user" &&
computer.controlLeaseId &&
computer.controlLeaseExpiresAt &&
computer.controlLeaseExpiresAt.getTime() > now.getTime(),
);
// Keep an active same-bot user lease so Skip / I'm done appear immediately.
// Preserve another bot's active lease — computer/takeover owns revocation. Do not attach
// this run to that lease: release validates controlBotId and could clear the binding without
// resuming the waiting run. The requesting bot's takeover flow binds it after revocation.
// Otherwise clear stale control, but bind controlRunId so takeoverRequested is true.
const retainControl = Boolean(activeUserLease && computer?.controlBotId === input.botId);
const preserveForeignLease = Boolean(
activeUserLease && computer?.controlBotId && computer.controlBotId !== input.botId,
);
const marked = await tx.computer.updateMany({
where: { id: input.computerId, spaceId: input.spaceId },
data: retainControl
? { state: "running", controlRunId: input.runId }
: preserveForeignLease
? { state: "running" }
: {
state: "running",
controlHolder: "none",
controlLeaseId: null,
controlLeaseExpiresAt: null,
controlBotId: null,
controlRunId: input.runId,
},
});
if (marked.count !== 1) throw new Error("Computer was not available to mark for takeover");
const waitingEvent = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: "computer.takeover.requested",
runId: input.runId,
payload: {
reason: input.reason,
takeoverRequested: true,
retainedControl: retainControl,
},
});
await tx.event.deleteMany({ where: { runId: input.runId, type: "thread.progress" } });
return { threadId: waitingEvent.threadId, seq: waitingEvent.seq };
});
if (!committed) return false;
await notifyRealtime(realtime, committed.threadId, committed.seq);
return true;
}
export async function finalizeComputerControlRelease(
prisma: PrismaClient,
input: FinalizeComputerControlReleaseInput,
realtime?: RealtimeFanout,
): Promise<FinalizeComputerControlReleaseResult | false> {
const committed = await prisma.$transaction(async (tx: Prisma.TransactionClient) => {
const cleared = await tx.computer.updateMany({
where: {
id: input.computerId,
spaceId: input.spaceId,
controlBotId: input.botId,
controlLeaseId: input.leaseId,
controlRunId: input.runId,
},
data: {
controlHolder: input.holder,
controlLeaseId: null,
controlLeaseExpiresAt: null,
controlBotId: null,
controlRunId: null,
},
});
if (cleared.count !== 1) return null;
const checkpoint =
input.reason === "skipped" || input.reason === "expired" ? "takeover-skipped" : "takeover";
const resumed = input.runId
? await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
botId: input.botId,
status: "waiting_takeover",
},
data: {
status: "queued",
checkpoint,
},
})
: { count: 0 };
// A steered continue may already hold the run as leased/running. Stamp the
// checkpoint without stealing the lease so the worker can restore tools.
const stamped =
input.runId && resumed.count !== 1
? await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
botId: input.botId,
status: { in: ["leased", "running"] },
},
data: { checkpoint },
})
: { count: 0 };
const runId = resumed.count === 1 || stamped.count === 1 ? input.runId : null;
const bot = await tx.bot.findFirst({
where: { id: input.botId, spaceId: input.spaceId },
select: { thread: { select: { id: true } } },
});
if (!bot?.thread) return { threadId: null, seq: null, runId };
const event = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: bot.thread.id,
botId: input.botId,
runId: runId ?? undefined,
type: "computer.takeover.released",
payload: {
holder: input.holder,
leaseId: input.leaseId,
reason: input.reason,
},
});
return { threadId: event.threadId, seq: event.seq, runId };
});
if (!committed) return false;
if (committed.threadId && committed.seq !== null) {
await notifyRealtime(realtime, committed.threadId, committed.seq);
}
return { runId: committed.runId };
}
export async function appendEvent(
prisma: PrismaClient,
input: AppendEventInput,
realtime?: RealtimeFanout,
): Promise<ProductEvent> {
const event = await prisma.$transaction((tx: Prisma.TransactionClient) =>
appendEventInTransaction(tx, input),
);
const productEvent = mapProductEvent(event);
await notifyRealtime(realtime, event.threadId, event.seq);
return productEvent;
}
export async function finalizeRun(
prisma: PrismaClient,
input: FinalizeRunInput,
realtime?: RealtimeFanout,
): Promise<FinalizeRunResult | false> {
const committed = await withTransactionRetry(() => finalizeRunOnce(prisma, input));
if (!committed) return false;
await notifyRealtime(realtime, committed.threadId, committed.seq);
return { continuationRunId: committed.continuationRunId };
}
/** Stamps one turn-level wall-clock duration on the final tool block. */
export function completedRunBlocks(
blocks: MessageBlock[],
startedAt: Date | null,
completedAt: Date,
): MessageBlock[] {
if (!startedAt) return blocks;
const durationMs = completedAt.getTime() - startedAt.getTime();
if (!Number.isFinite(durationMs) || durationMs < 0) return blocks;
const index = blocks.findLastIndex((block) => block.kind === "steps");
if (index < 0) return blocks;
return blocks.map((block, blockIndex) =>
blockIndex === index && block.kind === "steps"
? { ...block, durationMs: Math.round(durationMs) }
: block,
);
}
async function finalizeRunOnce(
prisma: PrismaClient,
input: FinalizeRunInput,
): Promise<{ threadId: string; seq: number; continuationRunId: string | null } | null> {
return prisma.$transaction(async (tx: Prisma.TransactionClient) => {
await tx.$queryRaw`SELECT id FROM threads WHERE id = ${input.threadId} FOR UPDATE`;
let writableRun: { startedAt: Date | null } | undefined;
try {
writableRun = await assertRunCanWriteHistory(tx, input.runId);
} catch (error) {
if (error instanceof RunHistoryWriteError) return null;
throw error;
}
const now = new Date();
const terminal = await tx.run.updateMany({
where: {
id: input.runId,
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
taskId: input.taskId,
status: "running",
leaseOwner: input.leaseOwner,
leaseFence: input.leaseFence,
},
data: {
status: input.outcome,
error: input.outcome === "failed" ? input.error : null,
completedAt: now,
leaseOwner: null,
leaseExpiresAt: null,
toolCallBudgetCount: null,
toolCallBudgetExceeded: null,
},
});
if (terminal.count !== 1) return null;
const attempt = await tx.attempt.updateMany({
where: {
id: input.attemptId,
runId: input.runId,
fence: input.leaseFence,
status: "running",
},
data: {
status: input.outcome,
error: input.outcome === "failed" ? input.error : null,
finishedAt: now,
},
});
if (attempt.count !== 1) throw new Error("Active run attempt was not available to finalize");
const task = await tx.task.updateMany({
where: {
id: input.taskId,
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
},
data: { status: input.outcome },
});
if (task.count !== 1) throw new Error("Run task was not available to finalize");
if (input.outcome === "completed") {
const completedBlocks = completedRunBlocks(input.blocks, writableRun?.startedAt ?? null, now);
if (completedBlocks.length > 0) {
const message = await createThreadMessageInTransaction(tx, {
threadId: input.threadId,
role: "bot",
blocks: completedBlocks,
botId: input.botId,
runId: input.runId,
markUnread: input.markUnread,
});
await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: "thread.message.created",
runId: input.runId,
payload: { messageId: message.id, role: "bot", blocks: completedBlocks },
});
}
}
const lastEvent = await appendEventInTransaction(tx, {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
type: input.outcome === "completed" ? "run.completed" : "run.failed",
runId: input.runId,
payload: input.outcome === "completed" ? {} : { error: input.error },
});
await tx.event.deleteMany({ where: { runId: input.runId, type: "thread.progress" } });
if (input.outcome === "completed") {
await tx.steeringMessage.deleteMany({
where: { runId: input.runId, claimedAt: { not: null } },
});
await tx.steeringMessage.updateMany({
where: { runId: input.runId },
data: { runId: null },
});
} else {
const { sourceMessage } = await tx.run.findUniqueOrThrow({
where: { id: input.runId },
select: { sourceMessage: { select: { seq: true } } },
});
// A continuation's source is the newest steering it was created for. Only newer
// messages justify another run after failure, even if setup failed before claiming.
await tx.steeringMessage.updateMany({
where: {
runId: input.runId,
message: sourceMessage ? { seq: { gt: sourceMessage.seq } } : undefined,
},
data: { runId: null },
});
}
const continuationRunId = await createSteeringContinuation(tx, input);
await tx.bot.update({ where: { id: input.botId }, data: { updatedAt: now } });
return { threadId: lastEvent.threadId, seq: lastEvent.seq, continuationRunId };
});
}
async function createSteeringContinuation(
tx: Prisma.TransactionClient,
input: FinalizeRunBase,
): Promise<string | null> {
const active = await tx.run.findFirst({
where: {
threadId: input.threadId,
botId: input.botId,
status: { in: ["queued", "leased", "running", "waiting_input", "waiting_takeover"] },
},
select: { id: true },
});
if (active) return null;
const pending = await tx.steeringMessage.findMany({
where: {
botId: input.botId,
runId: null,
message: { threadId: input.threadId },
},
include: { message: { select: { id: true, blocks: true, seq: true } } },
orderBy: [{ message: { seq: "asc" } }, { id: "asc" }],
});
if (pending.length === 0) return null;
const last = pending.at(-1)!;
const task = await tx.task.create({
data: {
spaceId: input.spaceId,
botId: input.botId,
threadId: input.threadId,
userId: pending[0]!.userId,
prompt: "Respond to the user's steering context.",
status: "queued",
},
});
const run = await tx.run.create({
data: {
spaceId: input.spaceId,
botId: input.botId,
threadId: input.threadId,
taskId: task.id,
userId: pending[0]!.userId,
status: "queued",
trigger: "follow_up",
sourceMessageId: last.message.id,
},
});
await tx.steeringMessage.updateMany({
where: { id: { in: pending.map((item) => item.id) }, runId: null },
data: { runId: run.id, claimedAt: null },
});
return run.id;
}
export async function appendEventInTransaction(
tx: Prisma.TransactionClient,
input: AppendEventInput,
) {
const thread = await tx.thread.update({
where: { id: input.threadId },
data: { nextEventSeq: { increment: 1 } },
select: { nextEventSeq: true },
});
await assertRunCanWriteHistory(tx, input.runId);
// Unpaired UTF-16 surrogates (e.g. a split emoji high half) are invalid JSON for Postgres.
const payload = sanitizeJsonValue(input.payload);
return tx.event.create({
data: {
spaceId: input.spaceId,
threadId: input.threadId,
botId: input.botId,
seq: thread.nextEventSeq - 1,
type: input.type,
payload: payload as Prisma.InputJsonValue,
runId: input.runId,
},
});
}
async function notifyRealtime(
realtime: RealtimeFanout | undefined,
threadId: string,
cursor: number,
): Promise<void> {
await realtime?.publish(threadTopic(threadId), JSON.stringify({ cursor })).catch(() => undefined);
}
export async function eventsAfter(
prisma: PrismaClient,
threadId: string,
cursor: number,
limit?: number,
) {
return prisma.event.findMany({
where: { threadId, seq: { gt: cursor } },
orderBy: { seq: "asc" },
...(limit ? { take: limit } : {}),
});
}
export async function* followThreadEvents(
prisma: PrismaClient,
threadId: string,
cursor: number,
realtime?: RealtimeFanout,
signal?: AbortSignal,
catchUpMs = realtime ? PUSH_CATCH_UP_MS : POLL_ONLY_CATCH_UP_MS,
): AsyncGenerator<ProductEvent> {
let seq = cursor;
const latch = new ChangeLatch();
const unsubscribe = realtime
? await realtime
.subscribe(threadTopic(threadId), () => latch.notify())
.catch(() => async () => {})
: async () => {};
try {
while (!signal?.aborted) {
const observedGeneration = latch.generation;
let batchSize = 0;
do {
const events = await eventsAfter(prisma, threadId, seq, EVENT_BATCH_SIZE);
batchSize = events.length;
for (const event of events) {
seq = event.seq;
yield mapProductEvent(event);
}
} while (batchSize === EVENT_BATCH_SIZE && !signal?.aborted);
if (signal?.aborted) break;
await latch.waitForChange(observedGeneration, catchUpMs, signal);
}
} finally {
await unsubscribe();
}
}
function threadTopic(threadId: string): string {
return `thread:${threadId}`;
}
function mapProductEvent(event: {
id: string;
spaceId: string;
threadId: string;
botId: string;
seq: number;
type: string;
payload: unknown;
runId: string | null;
createdAt: Date;
}): ProductEvent {
return {
id: event.id,
spaceId: event.spaceId,
threadId: event.threadId,
botId: event.botId,
seq: event.seq,
type: event.type as ProductEvent["type"],
runId: event.runId ?? undefined,
createdAt: event.createdAt.toISOString(),
payload: event.payload as Record<string, unknown>,
};
}
class ChangeLatch {
generation = 0;
private wake: (() => void) | undefined;
notify(): void {
this.generation += 1;
this.wake?.();
}
async waitForChange(expected: number, timeoutMs: number, signal?: AbortSignal): Promise<void> {
if (this.generation !== expected || signal?.aborted) return;
await new Promise<void>((resolve) => {
let settled = false;
const finish = () => {
if (settled) return;
settled = true;
clearTimeout(timer);
signal?.removeEventListener("abort", finish);
if (this.wake === finish) this.wake = undefined;
resolve();
};
const timer = setTimeout(finish, timeoutMs);
this.wake = finish;
signal?.addEventListener("abort", finish, { once: true });
if (this.generation !== expected || signal?.aborted) finish();
});
}
}